Any Computer / networking gurus here?

For things that don't fit into a category. i.e. non-motor related subjects.

Moderator: Forum Moderators

Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Any Computer / networking gurus here?

Post by Fordimus Prime »

I have a bit of a technical enquiry, any help would be much appreciated.

Are there any newtorking solutions that are faster than LAN, but dont cost as much as fibre channel?

I work for a small production company- we have large volumes of data, and need all our video data in one place, so multiple end suites can work off the same source video files. At the moment, we have a couple of large raid arrays hooked to one machine, and other machines access this via the LAN network- but editing HD, LAN just isnt fast enough to keep up, and everything bogs down.

Most solutions ive come across are waaaaay out of budget.
- MG ZR td+ -
empsburna
RT BiKiloPoster
Posts: 2727
Joined: Sun Jan 18, 2004 9:53 pm
feedback: 741225
Location: The Black Country

Re: Any Computer / networking gurus here?

Post by empsburna »

I build and secure large corporate networks for a living.

Need to know what kind of architecture you have in place at the moment.

Lighting up some fibre isn't as expensive as you might think, but you still need decent switch gear/cards/cables at each end to make any use of it.

What kind of distances are you thinking of covering?
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

are you using gigabit lan? as apposed to 100mbit?
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

Scott wrote:are you using gigabit lan? as apposed to 100mbit?
yup, we're on gigabit.

empsburna wrote:I build and secure large corporate networks for a living.

Need to know what kind of architecture you have in place at the moment.

Lighting up some fibre isn't as expensive as you might think, but you still need decent switch gear/cards/cables at each end to make any use of it.

What kind of distances are you thinking of covering?

The details: We run in a small office- distance would never be more than 5-10 metres. At the moment, we have highest category lan cables hooking everything up via a gigabit switch.

Currently with have 4 macpros, and a couple of drobo's for storage, which are hooked up to one macpro. In the past, each editor would be working on a single project, so projects would be transfered to each machine's video drive and edited locally- now we're doing bigger projects, where everyone needs access to the same data- so we need all the data in one place, and the ability for every machine to access the data without slowing everyone else down.

- so in to sum up, we need an expandable storage solution, to which multiple machine need very fast access to.

Most of the companys ive spoken to who set up edit suites, have only given me quite expensive options! Budget is between 5 - 10 k, depending on what the pros/ cons are.
- MG ZR td+ -
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

hmmm, i might be talking crap, but...

id be more inclined to work out a way to split the files and work on them locally still?

how can 2 people really be in the same 'file' for editing purposes? The software that your using, is there not a larger enterprise edition which can help facilitate this for example? Also, the drobo's have you done any benchmarking to see if they are actually fast enough? You might find that even though they are in a raid inside the drobo, if they only link to the mac by firewire 400 for exmaple, than that 1gbit lan isnt even half way to being utilised...
daibill
Forum Regular
Posts: 152
Joined: Thu Nov 09, 2006 12:34 am
Location: South Wales

Re: Any Computer / networking gurus here?

Post by daibill »

I'm with scott on this. Multiple users can't edit the same part of a video file at the same time. File will be locked open to whoever is using it.
If it were to be sectioned out into smaller chunks and worked on locally, then your in with a shout using gigabit. Once everyone is happy then put the pieces together then store it.

Are you using a managed or un-managed switch?
How large are the files?
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

sorry guys- please, i know what im talking about, and i know what my company needs

Say we've filmed an exhibition- one editor needs to edit the main video, the other one needs to work on a short 'advert style video'. Both need to use the same video files. final cut is non destructive- very much so that multiple editors can work on the same video footage. both editors are working in different projects, but need access to the same video files.

These files are huge. 1 tape is about 10 gig. Shifting and copying things around wouldnt work and would get messy.

There are systems in the broadcast industry for this. Final cut server is a server based program that you can have on an xserve raid- oranganises all the files, and communicates with final cut and prevents one editor from opening a final cut pro file another editor is working on.

Like ive said, what we need, is an upgradable storage solution, and a very fast network. but i dont think its realistic with the budget
- MG ZR td+ -
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

daibill wrote:I'm with scott on this. Multiple users can't edit the same part of a video file at the same time. File will be locked open to whoever is using it.
do you work in a production company? Multiple users can easily edit the same file. Most modern editing programs are non destructive - they dont change the source file in any way. You dont open a movie file and 'edit it'. You create a project, and put movie files into that project. the movie files themselves are never altered, just referenced to.

you could have 10 computers editing 10 versions of the same footage- only limit would be network speed and hard drive speed.
- MG ZR td+ -
Cableguy
Silver Trader
Silver Trader
Posts: 22323
Joined: Mon Jun 14, 2004 11:21 am
feedback: 311135
Location: Near Junction 9 M6

Re: Any Computer / networking gurus here?

Post by Cableguy »

Yeah this software works the same as things like lightroom , the source is used as a reference and effectively a local copy is made for editing before exporting.

If your using cheapo gigabit swithces then it will bog down , but a properly setup gigabit router/switch setup from decent server should be able to supply relatively fast, 10gb is quite hefty for real time access though.

What make and model of gigabit switch do you have currently ?
Image
The Storm Has Gone Again
Bubble/Mk3 conversion looms available including wasted spark AND ZS, pm for prices !
daibill
Forum Regular
Posts: 152
Joined: Thu Nov 09, 2006 12:34 am
Location: South Wales

Re: Any Computer / networking gurus here?

Post by daibill »

Sounds like you are going to have to hit this with a very big bag of money..
daibill
Forum Regular
Posts: 152
Joined: Thu Nov 09, 2006 12:34 am
Location: South Wales

Re: Any Computer / networking gurus here?

Post by daibill »

Fordimus Prime wrote:
daibill wrote:I'm with scott on this. Multiple users can't edit the same part of a video file at the same time. File will be locked open to whoever is using it.
do you work in a production company?

Not at all mate. I'm a senior network engineer. CCNA and MCSE
empsburna
RT BiKiloPoster
Posts: 2727
Joined: Sun Jan 18, 2004 9:53 pm
feedback: 741225
Location: The Black Country

Re: Any Computer / networking gurus here?

Post by empsburna »

What kit do you have in place at the moment.

I would like to have a look at the throughput at both ends (server and client).

managed or dumb switches?

For the distances you are talking about you should have very little loss.
IrishRover
Rovertech Support Team
Rovertech Support Team
Posts: 8105
Joined: Tue Sep 16, 2003 4:16 am
feedback: 1050163
Location: Leinster, Republic of Ireland
Contact:

Re: Any Computer / networking gurus here?

Post by IrishRover »

I'd say it's the Drobos which are slowing things down. I have read that they are quite a bit slower than a normal raid array would be (maybe something to do with their internal disk virtualisation.
You'd probably find a cheap server from HP or Dell with two to four SATA disks and a RAID controller card is faster.
Try giving HP or Dell a call and tell them what you want and why you need it.
Here's a few links to start off:
http://h10010.www1.hp.com/wwpc/uk/en/sm ... 83705.html
http://configure.euro.dell.com/dellstor ... 1051&s=bsd

You could configure either of those with 4G RAM a raid controller card with 2 SATA disks of 750GB storage for less than £1500

To answer your question literally though, there is something faster than gigabit LAN and cheaper than 10 gigabit fibre. USB3 will be available next year and should run at speeds of 4.8 gigabit (600 megabytes per second transfer speed). It should be possible to network devices over USB, but I haven't experimented with it myself.
Image
2x 216 GTi DOHC
3x 220 Coupe Turbo
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

Cableguy wrote:Yeah this software works the same as things like lightroom , the source is used as a reference and effectively a local copy is made for editing before exporting.

If your using cheapo gigabit swithces then it will bog down , but a properly setup gigabit router/switch setup from decent server should be able to supply relatively fast, 10gb is quite hefty for real time access though.

What make and model of gigabit switch do you have currently ?
Yeah, fcp kind of works like that. when viewing the file in the edit program, your watching the original- when you make changes, its rendered to a render file locally. When you export the final version, it'll use a mixture of render files and the original to construct the final export. Since the original is only ever read from, and is never changed, as many people as you like can edit from the original video.

regarding our router / switch

we're using a NETGEAR GS608 8-Port Gigabit Desktop Switch.

trouble is, we dont just need it a little faster- we need seriously faster, and at a high sustained data rate.

Its a shame we cant have a raid system that can connect to all the computers via firewire 800 or esata- but those protocols arent designed for networking.

My options so far to the boss are - spend a lot of money, or give the editors more time, so we dont work on multiple projects at the same time!

I keep wondering why there isnt a budget solution, but then there arent many situations where youd need such high data transfer rates. Probably only video production companies, and biiig companies with a serious amount of networking going on.
- MG ZR td+ -
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

IrishRover wrote:I'd say it's the Drobos which are slowing things down. I have read that they are quite a bit slower than a normal raid array would be (maybe something to do with their internal disk virtualisation.
You'd probably find a cheap server from HP or Dell with two to four SATA disks and a RAID controller card is faster.
Try giving HP or Dell a call and tell them what you want and why you need it.
Here's a few links to start off:
http://h10010.www1.hp.com/wwpc/uk/en/sm ... 83705.html
http://configure.euro.dell.com/dellstor ... 1051&s=bsd

You could configure either of those with 4G RAM a raid controller card with 2 SATA disks of 750GB storage for less than £1500

To answer your question literally though, there is something faster than gigabit LAN and cheaper than 10 gigabit fibre. USB3 will be available next year and should run at speeds of 4.8 gigabit (600 megabytes per second transfer speed). It should be possible to network devices over USB, but I haven't experimented with it myself.

thanks for looking into that, but the drobos arent the bottle neck (at the moment anyway)- the network speed is. The drobo is hooked up via firewire 800, and is plenty fast enough at the moment, for that once computer hooked up to it. But its all the other machines accessing it through network that bog down- Even if just one computer is accessing footage, if that computer is doing so through the network, it can struggle. Heavy edits involve split screen etc, so can involve multiple video sources, so the network has to transfer a lot of data.

XDcam HD video fromat has a bit rate of 35 Mbits a sec. So 2 or 3 of those a sec over firewire 800 isnt a prob. But squeezing that data flow into ethernet is

A do agree with you though, those drobos arent quite designed for the job, but as a storage solution they worked out well (up until now anyway). We're running 4 terrabytes in each one! and we still need more space really. Always the way- you always need more speed, more hard drive space, more processing power.....


whatve I done? Ive turned rovertech into a tech help line! (please, just dont tell me to turn in off and on again....:P )

i do appreciate everyones input by the way :thumbup:
- MG ZR td+ -
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

have you got the spec of the drobo boxes?

The problem will occur when multiple people are referencing the same data at differnt points through the large file on the same disk.

Even in raid, you may not be able to get the data fast enough from the drives. By all referencing one large file in multiple locations inside that file, a hard disk system is going to struggle. is it deffo raid 0 setup?

an analogy would be could you replace 2 car wheels in 30 mins, the answer might be yes, but not if you had to do it 1 stage at a time, at 2 different cars at different ends of the street, running back and forth between each step (read operation).

setting larger block size on the raid 0 might help. Raid 0 will help with overall throughput, for sequential reads, buts im not sure how much better it would be for multiple random reads at the same time... going to 4 disks and a larger stripe might help.

I bet if you shoved a SSD into the drobo box, you would notice huge increase in performance, as there is no huge time penalty from doing random reads...

Also, if you go the fileserver route, then you can have multiple NICS inside the server, and that way you will stop caining the hub/switch...
IrishRover
Rovertech Support Team
Rovertech Support Team
Posts: 8105
Joined: Tue Sep 16, 2003 4:16 am
feedback: 1050163
Location: Leinster, Republic of Ireland
Contact:

Re: Any Computer / networking gurus here?

Post by IrishRover »

The drobo isn't really raid anything Scott. It uses proprietary internal software to create a type of virtual storage system out of the disks you put into it.
Image
2x 216 GTi DOHC
3x 220 Coupe Turbo
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

i think i know the things, about 250quid, plug in upto 4 drives, have a big network drive...

the drobo is deffo the sticking point here in my eyes. regardless of network through put, which should be ample (unless your getting 5 users+ simultaneously). It will be the drives. you know , like copying a directory of files, and doing a defrag at the same time, on a small network drive sharing thing, it aint going to be happy.

Perhaps get an SSD, and stick it in the drobo for a test, copy 1 main project source file to it as required when your all going to use it at once. Assuming the drobos arent complete crap, you should get much better performance.

even though the drobos are on firewire800, it doesnt mean it automatically works at 800mbps, the drives will still only be doing 120mb/s or similar on a single read, and perhaps only 2x 20mb/s on multiple reads.
IrishRover
Rovertech Support Team
Rovertech Support Team
Posts: 8105
Joined: Tue Sep 16, 2003 4:16 am
feedback: 1050163
Location: Leinster, Republic of Ireland
Contact:

Re: Any Computer / networking gurus here?

Post by IrishRover »

Yes I would still suspect the drobo too, but perhaps not necessarily because of a deficiency in itself, but the fact that the drobo is being shared over a few different connections. First, there's the USB to computer, then the computer passes it on to the network and the network switch (I hope you have a switch, not just a hub!) passes it to one or more other computers. The USB is supposed to be very "bursty" for transferring data (chunks at a time with a lag in between) and then you have the computer it is attached to acting as a bridge, which may have any kind of software installed on it which might tie up the transfer through it to the network interface.

A direct NAS (network attached storage) solution would be best, but of course it can be expensive.
That's why I think a dedicated server with customised RAID controlled storage attached directly to the network switch would be faster.
Image
2x 216 GTi DOHC
3x 220 Coupe Turbo
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

ah, i missed the part that showed the drobo being connected to the mac, i assumed it was on the LAN.

Im guessing (for no real reason other than being macs) that they cant directly access the drives if the drobo was on the LAN? At least that way, the macpro isnt acting as a hub initself, which would slow things down...
Cableguy
Silver Trader
Silver Trader
Posts: 22323
Joined: Mon Jun 14, 2004 11:21 am
feedback: 311135
Location: Near Junction 9 M6

Re: Any Computer / networking gurus here?

Post by Cableguy »

Drobo doesnt have a network connection , so neg.
Image
The Storm Has Gone Again
Bubble/Mk3 conversion looms available including wasted spark AND ZS, pm for prices !
empsburna
RT BiKiloPoster
Posts: 2727
Joined: Sun Jan 18, 2004 9:53 pm
feedback: 741225
Location: The Black Country

Re: Any Computer / networking gurus here?

Post by empsburna »

Is this just an apple/OSX solution you need?

You are going to have issues if it is connected via a macbook with USB.
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

Scott wrote:ah, i missed the part that showed the drobo being connected to the mac, i assumed it was on the LAN.

Im guessing (for no real reason other than being macs) that they cant directly access the drives if the drobo was on the LAN? At least that way, the macpro isnt acting as a hub initself, which would slow things down...
We could hook the brobo up to the lan via an optional accessory for the drobo, but we didnt see the benefit- fw 800 has great sustained data rates, so the main drobo is hooked up like this to our number one edit suite, for when doing a really heavy project that needs maximum data rates.

IrishRover wrote:Yes I would still suspect the drobo too, but perhaps not necessarily because of a deficiency in itself, but the fact that the drobo is being shared over a few different connections. First, there's the USB to computer, then the computer passes it on to the network and the network switch (I hope you have a switch, not just a hub!) passes it to one or more other computers. The USB is supposed to be very "bursty" for transferring data (chunks at a time with a lag in between) and then you have the computer it is attached to acting as a bridge, which may have any kind of software installed on it which might tie up the transfer through it to the network interface.

A direct NAS (network attached storage) solution would be best, but of course it can be expensive.
That's why I think a dedicated server with customised RAID controlled storage attached directly to the network switch would be faster.
we're not using usb - usb is evil! - i know what you mean about it being bursty- its why all our HD's are hooked up by fw 800- firewire was designed for sustained data rates, and i think it was actually designed with digital video in mind.


I still think ethernet just isnt fast enough. even if i just connect to another edit suites video drive, and edit a project off there, it can get laggy, if theres lots of split screen.


i think i should give up being helpful. this is the editors problem, im supposed to just be a cameraman!
- MG ZR td+ -
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

oh how stupid- the drobos network attachment only connects via usb! thats probably why we didnt go that route. that introduces a bottleneck right there.

The reason we went for getting a couple of drobos was that it allowed us to expand our storage quite easily- other raid arrays, when a drive fails you have to replace it with exactly the same size drive.

With the drobo, say it has 4 x 500 gig hds, its possible to eject one drive replace with a 1.5 terrabyte drive, let it rebuild. then do the same for next drive, then rebuild, and so on for all the others. You keep all your data, but youve hugely expanded your storage. Oh and the other reason we liked them was they were relatively cheap at the tie compared to other raid type systems.
- MG ZR td+ -
IrishRover
Rovertech Support Team
Rovertech Support Team
Posts: 8105
Joined: Tue Sep 16, 2003 4:16 am
feedback: 1050163
Location: Leinster, Republic of Ireland
Contact:

Re: Any Computer / networking gurus here?

Post by IrishRover »

Oh yes, I also missed the bit where you said firewire and not usb!

I did read that there was a network interface optional accesory out for the Drobo now. I suspect that even though you can now connect directly to the network (and eliminate the "bridge computer" from the equation), with this optional Droboshare kit, you still have an extra stage in the way with the Drobo to Droboshare and then the Droboshare to the network switch.
I can't see how the Drobo connects to the Droboshare here, but I guess it is via USB.
http://www.drobo.com/Products/droboshare.php
Maybe you can connect the two with a firewire cable and it would give a better performance, but I don't can't see if this feature is offered.
I supposed for another $200 or whatever it costs in the UK, it would be a cheap experiment to see if it speeds up things by eliminating the computer to which it is directly attached at the moment.

[EDIT]
Yeah I see above now you've confirmed it is only USB from Drobo to Droboshare. Still, I suppose if you have a budget of £10,000 to improve things, it might be worth a punt of £200 to see if the Droboshare does improve things, despite being a USB to network connection.
Image
2x 216 GTi DOHC
3x 220 Coupe Turbo
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

I recon that the drobo is still the issue, multiple reads are not going to be its strong point.

by the sounds of it, rebuilding a drive when a larger drive is put in doesnt to me indicate raid 0, so your not getting multiple drive speed/benefit. Still, even with multiple drives in a raid 0, your not going to get good multiple reads necessarily, which appears to be the problem (to me at least).

there should be plenty of overhead on 1000mbit/sec network to transport around 30mbit/sec streams... seeking through the files might be an issue though, as i believe jumping through avi's mean that all frames have to be decoded inbetween (hence needing to pull huge data).
Null_Byte
RT GOD
Posts: 11339
Joined: Tue Sep 21, 2004 11:47 pm
feedback: 465046
Location: today i will be mostly... under the bonnet
Contact:

Re: Any Computer / networking gurus here?

Post by Null_Byte »

Hmm

Ok so the drobo box is connected via FW, and the other Macs are connecting to the one with the drobo box, that isn't a very efficient way to do it at all.

You will run into the limitations of the MAC you are transfering the data through - i am doubtful the problem is your network.

I think with a proper managed gigabit switch, something like a decent procurve and some form of network attached storage, or a dedicated server would see you with better performance.

Most decent RAID setups will allow you to use disks of different sizes. With the budget you have you should be able to put together something quite easily.

Either that or look at a 2nd hand SAN.
Punx0r
Rovertech Moderator
Rovertech Moderator
Posts: 32552
Joined: Mon Nov 29, 2004 8:42 pm
feedback: 538363
Location: Northants

Re: Any Computer / networking gurus here?

Post by Punx0r »

Much of this is over my head, but a lifetime of buying HDD's suggests to me that despite increasing interface speeds, ata133, sata1, sata2 etc, the internal drive speeds don't seem to have improved much. Generally considerably less than 100Mb/sec sustained transfer. As Scott says, multiple reads from the same drive will trash transfer speeds.
Anthony | 1997 800 Vitesse Coupe, 1985 SD1 Vitesse
Scott
RT GOD
Posts: 13516
Joined: Sun Jul 06, 2003 9:36 pm
feedback: 210998
Location: No longer in a rover in oxfordshire :(
Contact:

Re: Any Computer / networking gurus here?

Post by Scott »

well a raid will help with 1 single read , larger caches , and higher areal density, and clever caching (samsung spinpoint drives) work better, but multiple reads are the problem here as i see it.

let us know what you decide and if it works :P
Steve
RT BiKiloPoster
Posts: 2726
Joined: Fri Feb 21, 2003 9:22 am
feedback: 197496
Location: Tamworth

Re: Any Computer / networking gurus here?

Post by Steve »

I think you have a couple of issues.

The first being your Netgear switch. You need to take an in-depth look as to what the backplane speeds are of the switch (i.e. the speed at which the ports communicate). I would suggest something decent like a Procurve or Cisco

Also, you need a dedicated machine with dedicated storage. Depending on how much space you need, something like this may do:

http://h71016.www7.hp.com/ctoBases.asp? ... ype=Matrix

OR

This as a base server:

http://h10010.www1.hp.com/wwpc/us/en/sm ... 21486.html

Add in a P400 SAS controller with 512mb cache

and something like this as your storage connected via SAS:

http://h18006.www1.hp.com/storage/disk_ ... index.html

The beauty of the HP stuff is that when you team the NIC's, you end up with Large Send Offload meaning your TX path is 2Gbps and RX path is 1Gbps, so sharing the files between multiple editors should not be a problem.

Bear in mind that the highest performing, yet safest RAID is RAID10 (Stripped Mirror). To increase performance, you need to increase the count of spindles (hard disks) as you span the data over more spindles therefore decreasing how much each one stores thus making it quicker to read.

HTH

Steve
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

I absorbed a lot of info yesterday- you guys were pretty much on the ball.

After a chat to a few companies who specialize in network setups for production companies, we're going to be getting something like this:


Something like a 27 terrabyte raid array, a managed router, and one of our old G5 macpro to run the setup, and dual ethernet connections to all the edit suite macpros.

I havent got the details to hand, but the number work out that we'll get at least 4 times faster sustained data rates through the network.

As you all mentioned, the bottle necks in our current ghetto setup, are the cheapo switch, which is more for home networking rather than high speed, and the drobo into one mac, then network to the others.

Once it was explained to me, it seemed so obvious- say 2 macs put in a command for some data. one mac is directly connected to the drobo, the other goes through the network to the other mac, then to the drobo.

Drobo gets to commands for data, and dishes it out in what it thinks is the most efficent way to one mac. That mac tries to then send it across the network in what it thinks is the best way (meanwhile its now doing things with its own data, which it'll give priority to). Then the switch, which is dealing with all other network activity, also tries to route the information it what it thinks is the best way. Bottle neck after bottle neck, causing quite bursty and slow network speeds

So a managed switch directly connected to the raid array, and directly connected to each computer via dual ethernet = much better. the switch manages everything, and you get mega fast networking.

simples!

The raid array at 27 terrabytes is a bit overkill, but im told if you only half fill it, then want to upgrade, the data has to come off the unit, then put more drives in, then put data back. But if you fill the array, if you want to upgrade in the future, you can just add another blade.

i think we've been quoted 8 grand for the entire setup, including a couple of days to set it up (apparently its a real bugger to get set up correctly) and it includes an on site maintenance thing, so they'll come and fix any problems.

Now we know whats possible, we're going to shop around a bit.

I've learnt so much about networking in the past few days! My hat goes off to anyone who has to deal with this on a day to day basis!
- MG ZR td+ -
Steve
RT BiKiloPoster
Posts: 2726
Joined: Fri Feb 21, 2003 9:22 am
feedback: 197496
Location: Tamworth

Re: Any Computer / networking gurus here?

Post by Steve »

Just another thing to consider...Backups.

27Tb is a LOT of data and I would hate to loose the array's for some reason. Make sure you get a good, decent backup solution in place.
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

Steve wrote:Just another thing to consider...Backups.

27Tb is a LOT of data and I would hate to loose the array's for some reason. Make sure you get a good, decent backup solution in place.
yup, id already made noises about that. We used to keep the original footage on tape, so all we needed to do was backup the edit file, and any graphics/ final exports etc to DVD. the actual footage was kept on tape, if we ever needed to restore a project we could just recapture the footage and load up the edit file and reconnect the media

But now we're starting to go tapeless.....

As a temp solution, once the raid is in place, we'll have 2 drobos going spare- at 4 tb big each, thats 8 terrabytes, which might do as a temporary backup solution.

I've heard of so many horror stories of tapeless production- so many people just arent used to the workflow. I heard a nasty story in america:

The edit house would usually recieve the tapes for whatever TV production they were working on. The editor would edit them. Tapes would be mailed to a grading studio the other side of town, for colour grading, and the editor would just email over the EDL (edit descion list) which in essence contains his edit, but none of the footage- tape would be recaptured at the grading house, and the edl loaded up.

No- one had told the editor that the project had been recorded on a tapeless camera. He emailed the edl over as normal, deleted all the footage from his computer, went onto the next project. A few days later the grading house called to ask where the tapes where......

a whole couple of episodes of a drama series where lost. They tried recovering the edit suits HD, but to no avail, he's already been working on other projects, and they could only recovery fragments.- same went for the cameras HD. It had to be a reshoot, and cost a lot of money
- MG ZR td+ -
Null_Byte
RT GOD
Posts: 11339
Joined: Tue Sep 21, 2004 11:47 pm
feedback: 465046
Location: today i will be mostly... under the bonnet
Contact:

Re: Any Computer / networking gurus here?

Post by Null_Byte »

No idea why you would need a router...

Personally I would use a decent quality managed switch, build a dedicated server (you could use one of the linux NAS optimised distros) dual gigabit cards (decent brand like 3com) trunk two lines to the switch then run your macs from that.

That way you aren't limiting yourself to pushing all your data through one mac.

For a backup solution you could hang the drobos, or any other storage medium NAS box, USB drives etc. off one Mac. Use Rsync to backup on a cron job. Rsync just copies the changes to the files over, so once you have made the initial backup, future backups will be swift.

You could even invest in a dual layer burner and archive off to DVD if you so wanted.

I would suspect this would be sufficient for the sort of system you require, if it isn't then you are going to either have to change workflow, or dig deep for a serious storage solution. SATA based SAN's are coming down in price.
Fordimus Prime
Rovertech Moderator
Rovertech Moderator
Posts: 2783
Joined: Tue May 04, 2004 6:58 pm
feedback: 332911
Location: middlesex
Contact:

Re: Any Computer / networking gurus here?

Post by Fordimus Prime »

Null_Byte wrote:No idea why you would need a router...

Personally I would use a decent quality managed switch, build a dedicated server (you could use one of the linux NAS optimised distros) dual gigabit cards (decent brand like 3com) trunk two lines to the switch then run your macs from that.

That way you aren't limiting yourself to pushing all your data through one mac.

For a backup solution you could hang the drobos, or any other storage medium NAS box, USB drives etc. off one Mac. Use Rsync to backup on a cron job. Rsync just copies the changes to the files over, so once you have made the initial backup, future backups will be swift.

You could even invest in a dual layer burner and archive off to DVD if you so wanted.

I would suspect this would be sufficient for the sort of system you require, if it isn't then you are going to either have to change workflow, or dig deep for a serious storage solution. SATA based SAN's are coming down in price.
i meant switch, not router, sorry- i always get the two mixed up.

What youve mentioned sounds very similar to the specs we've been quoted- the dual gigabit cards etc- i havent got the specs to hand as im off work for a month. (just hope they order the right setup while im gone!)

we were going to use an old mac as a server just for now as a cost saving measure. What i liked about the whole setup though was it sounded future proof- raid system all rack mounted, when we expand, dump the old mac and get a dedicated server slotted in there, then when we need more space, stick a other blade in

Will have a look at rsync - sounds ideal for our needs.

Think ive gone on holiday at just the right time - will get back, and (hopefully) everything will be in place and working!
- MG ZR td+ -
Post Reply

Return to “The Living Room (Posh man's lounge)”