Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

10 Gigabit Ethernet Suggestions: NAS and NIC

Eric Bowen said:
I normally just try and give people the information and let them decide for themselves. The 10Gb market though has really left very few choices though that are reasonable for small business. The market ie manufacturers are still trying to cash in on the fiber cow as long as possible even with Intel doing everything they can to push 10Gbe especially to small business. Netgear seems to be the only 1 with a clue who understands small business requires sub $5K and $2K switches while Intel and small tree realize adapters need to be sub $400. If they can put these 10gbe ports on $600 server boards that normally sell for $450 to $500 then they can lower the cost on 10Gbe hardware. They are just not doing it right now to push their Fiber cow and profit.
Click to expand...

As much as I rip on the NetGear ProSafe switches, I totally agree with everything you're saying here.

I'd love to see some upstart company walk in and turn the market on its head. No reason a 10GB 12 port switch can't be made to run silently and sell for under $350.

Hmmm.... Might have to put some real thought into this. No reason to limit to 10Gbps, either. I've seen lab demonstrations of over 80Gbps being pushed over 100 meters on regular Cat-6.


EDIT> A huge inflationary factor in the 10G Ethernet world are the licensing fees.
 
Last edited:
Please do. I really thought Netgear would finally start forcing the hand of the Big boys in the network industry with Intel behind them but their not. Gbit is basically as old as LCD display technology and the only reason it's lasted this long has been the average demand of Corp IT has not required major updates to bandwidth per client. The cost per port of 10Gbe would be down in the basement if manufacturing as a whole went to 10Gbe. Cat 7 has shown potential to easily push 40Gbs with Ethernet and still room to grow. The Media Content revolution unfolding hopefully will be the final push to drive the market as a whole to baseline build to 10gbe while tiering out 40Gbe or higher for high bandwidth demands. This however is another one of those markets that have lived far to fat on the hog at the expense of the business and consumer while the rest had to innovate technology advancements and efficiency in production.

The licensing fee's come down if the volume generated becomes global standardization. It's a numbers game and right now the numbers of ports generated are far to low.
 
Last edited:
PatrickFaith said:
I'm looking at the Cisco SG500XG-8F8T, anyone have any idea how loud it is ... will be in a vfx office area. I have had the same problem as Jeff K, with some switch manufactures being either extremely loud (arista is great but sounds like a turbine engine) or having limited buss speeds. I am looking for something under $6k. I only need 4 ports, 8 would be nice. I btw took Jeff's advice and got two ATTO thunder links.
Click to expand...

If you only need 4 ports, and are concerned about noise, just go infinband, and skip the switch entirely.
(4) Connect-x VPI dual port cards for $250 each, you can run 56gbs infiniband, or 40gbe IP over IB if you rather.
You can just run a subnet manager service that manages the IB frabric, you dont have to have a hardware switch.
With SMB 3.1 and RDMA performance is pretty much limited by the 8X PCIE3.0 slot or your Raid.

From Workstation to Workstation we get about 5500 MB/sec, with about 8-10% CPU overhead.

Most places are upgrading to the ConnectX-4 cards for 100gbe so you can pick up the older cards cheap.
 
Jeff Kilgroe said:
About average according to the specs. Sometimes it's not about dB levels, but the harmonics or pitch of the sound too. Many of us work in noise-sensitive environments and not all the equipment can be locked away in a sound proof room. Too many companies don't focus any efforts into noise control on their products and it's rather disheartening. Many of the switches I buy usually end up with voided warranties because I often yank out the fans to replace with an alternative solution in order to control noise levels....

I've used the 12 port Cisco in that series. Nice switch. The fans are not as high-pitched or whiney as the ProSafe ones. Still sounds like a vacuum cleaner more than the Arista turbine sound.
...
Click to expand...
Some of the frequencies drive me crazy, some of them surge/vary the fan speed for no apparent reason. My vfx room has 200 terabytes of disk spinning, so isn't the quietest place in the world. Also it's really important for the vfx guys for the networks not to hiccup since the batch jobs often run overnight. I wear boss q20 earplugs in that room so i'm not distracted from talking and other stuff, those earplugs work really well on rumbly pink noise ... so the cisco's sounds like the best fit for this small room right now. For this type of stuff, i'm the chasm adopter (i.e. not early adopter, but the shore line of users after the early adopter chasm is crossed) ... anyway so me finally buying this is probably a good indication for these companies to get their act in gear. Completely agree on the general incompetence on noise design with fans, I often complain to the vendors on that front and often tell them i will not buy their equipment (for large enterprise purchases) if the fan issue is not taken care of before it's shipped. I just don't understand how engineers design equipment and don't think about sound.


Christopher*Seguine said:
If you only need 4 ports, and are concerned about noise, just go infinband, and skip the switch entirely.
(4) Connect-x VPI dual port cards for $250 each, you can run 56gbs infiniband, or 40gbe IP over IB if you rather.
You can just run a subnet manager service that manages the IB frabric, you dont have to have a hardware switch.
With SMB 3.1 and RDMA performance is pretty much limited by the 8X PCIE3.0 slot or your Raid.

From Workstation to Workstation we get about 5500 MB/sec, with about 8-10% CPU overhead.

Most places are upgrading to the ConnectX-4 cards for 100gbe so you can pick up the older cards cheap.
Click to expand...
Didn't know about those, pretty sweet cards. For my vfx post room it is mainly macs with some scattered z840's, so kind of moving things to 10g just as a standard for everything. I do have another room i'm doing where a bunch of z840's with those cards would make a nice network, but for satellite offices i need brain dead network management which makes me a bit timid going IB since I have no experience on that.
 
Count me in as interested in all of this.

Synology's DS2015xs gets you 8 bays and two SFP+ 10gbE ports for $1380. I'm not sure if it's got enough power under the hood for what's needed, but it's an option.

Since this would be for my home use, I am a little interested in just direct-attaching it via SFP+ to a PC and a laptop and not worrying about 10g switches altogether until they make one that is quiet! Direct attach seems to work and they tested it here:
http://www.anandtech.com/show/8985/synology-ds2015xs-review-an-armbased-10g-nas/2

BTW, RE: SFP+ vs Cat-6 / RJ45, what do you smart folks think about that?

From what I understand, 10gbE over Cat-6 has higher latency and the chipsets run hotter than the SFP+ alternative. But on the other hand Cat-6 is more convenient and maybe good enough... but maybe I do not understand right :)

Ah man. I remember back when the Powerbook G4 came out with GigE and we were were amazed at the slim titanium GigE laptop! It's sad that you can't get a good workstation X99 motherboard with built-in 10gbE. Maybe someone will have the guts to do it when Broadwell-E comes out?

It kinda astounds me that for all of the supposed openness and flexibility in PC land, the manufacturers mostly just sell variations of the same stuff. Like, if HP wanted to make an actual workstation, you'd think they could make a motherboard with like 4 Thunderbolt ports, dual 10GbE, etc for what they charge. Sigh!

Well, I guess the upside is that we can build our own stuff - or buy from system builders like Eric - and have systems that beat HP and IBM and Dell, etc.

But yes, someone please build a cheap 10gbE switch!

Bruce Allen
www.boacinema.com
 
SFP+ is lower latency on average than Ethernet. However the transmission mode ie cabling, switches, and adapters are far more expensive on average. Also SFP+ has a very short range if you run copper. You have to run optical to get the longer ranges. Ethernet is really the best combination of all considerations. The latency is low enough. The cost to implement is far better since adapters, switches, and cable is far cheaper. The cable range is 100 meters on Cat 7 at 10Gbe which is often more than enough for small business infrastructure. Overall Ethernet is the best way to go on average.
 
 
Eric Bowen said:
SFP+ is lower latency on average than Ethernet. However the transmission mode ie cabling, switches, and adapters are far more expensive on average. Also SFP+ has a very short range if you run copper. You have to run optical to get the longer ranges. Ethernet is really the best combination of all considerations. The latency is low enough. The cost to implement is far better since adapters, switches, and cable is far cheaper. The cable range is 100 meters on Cat 7 at 10Gbe which is often more than enough for small business infrastructure. Overall Ethernet is the best way to go on average.
Click to expand...

Thank you, Eric!

Ah man, someone should seriously do a fan modification on those Netgear switches (or whatever other router Jeff likes). GROUP BUY! Silent 10g switch!

I am so obsessed with quiet that I have even been looking at those IsoBox quiet enclosures... damn expensive though!

Bruce Allen
www.boacinema.com
 
PatrickFaith said:
What's a good way of connecting a Cisco SG500XG-8F8T (i.e. 10g switch) to my normal 1G switch (i.e. my router and some of the computers are going to stay 1g, though all my servers and vfx workstations will cut over to 10g)?
Click to expand...

Sadly, you basically just have to connect one switch to the other and the link between the two will be 1G. If your 1G switch has a designated uplink port, use that to connect to one of the common 10G ports on the Cisco. There may be something else to do, but I'm guessing not since you didn't give any other info.

Ideally you would run a 1G switch that has one or two 10G uplink ports so that you can supply much greater bandwidth between your switches and connected hardware. Unfortunately, there are not too many such switches on the market, but they're slowly becoming more common as small unmanaged 1G switches with a 10G uplink are like the most logical and most requested type of switch these days.
 
Jeff Kilgroe said:
Sadly, you basically just have to connect one switch to the other and the link between the two will be 1G. If your 1G switch has a designated uplink port, use that to connect to one of the common 10G ports on the Cisco. There may be something else to do, but I'm guessing not since you didn't give any other info.

Ideally you would run a 1G switch that has one or two 10G uplink ports so that you can supply much greater bandwidth between your switches and connected hardware. Unfortunately, there are not too many such switches on the market, but they're slowly becoming more common as small unmanaged 1G switches with a 10G uplink are like the most logical and most requested type of switch these days.
Click to expand...

Thanks much Jeff, hooked it up like you said. Basically two mac pro's (each mac pro thunderbolt 2 to a atto thunder link, so i'm using two separate thunder links) then the thunder link connects to the cisco sg500xg. I then plug in my 1g router and another two 1g computers into the switch. I'm getting between 400 to 600 megabytes per second file transfer speeds over the network, which i think is pretty good. The switch has a soft rumbly sound, but after awhile doing high volume network stuff the blower kicks off and steps up a notch louder (not an anoying pitch, is more like a normal white noise, but when it steps up a notch it's hard to talk over). Going to hook my z840 into the network next. Is real nice having a decent low budget production 10g network. Again, thank you all for the help in this thread !!!
 
Can someone explain the "cable" options for 10g, I think I get the cat6 cables. it's the other cable's like when I connect to my hp z840. I don't get those long cable connectors, I ended up buying a cisco sfp-10g optical cable (it was like $150 for the cable). I also have a "copper" version coming. I completely don't get all the options I see on these cables (i.e. sfp, sfp+, wire gauges, etc), it's a entire list of stuff for each cable and I have no idea what is important. Also the prices for these cables is all over the place (i.e. is $50 cable all I need, or was the cisco $150 overkill). I am not even sure what I should call the "deep" connector ports on the switch, and why they are so deep.
 
Patrick how is the environment setup? Are you doing short distances or are there multiple edit suites spread out? The type of cables play a role in distance runs. For the most part, small single or two user environments work just fine with Cat 6/7/e.
 
Basically Fiber SFP+ copper has a far shorter distance limitation than Optical. Copper is cheaper though and requires lower power. The SFP+ 10Gbe adapters can have compatibility issues with certain SFP+ cables due to specs of the cable and the transmission built into the SFP+ adapters. If the 10Gbe network adapter is Intel then I suggest you get Intel SFP+ Twinax copper cables. Those adapters are very specific which cables they work with ie resistance and such. Ethernet overall though is cheaper and easier to deal with. 10Gbe Ethernet would be easier and cheaper to deal with if you have that option for the HP. I would suggest that since any Cat 6 or Cat 7 cable will work and distance options are better than copper SFP+.

http://www.cisco.com/c/en/us/produc...ransceiver-modules/data_sheet_c78-455693.html
 
Last edited:
10 Gigabit Feedback Loop

10 Gigabit Feedback Loop

Upgrading to 10 Gigbabit, just getting last feedback from the forum

@ Patrick F.: Any problems are hiccups with the Cisco sg500xg, happy with it?
@ Jeff K.: Any forthcoming switches that would preempt any of the current offerigs?
@ Anyone: For a Hackintosh build SmallTree vs ATTO vs Solarflare? Performance, price, stability?

Thanks in advance.

Sidebar: Have a Linksys Switch. I want to mod the fans to dampen the noise with a quieter fan such as Noctua. Doable or should I stay with stock fans for safety?
 
Jason Osterday said:
Patrick how is the environment setup? Are you doing short distances or are there multiple edit suites spread out? The type of cables play a role in distance runs. For the most part, small single or two user environments work just fine with Cat 6/7/e.
Click to expand...
So far everything is within 15 feet, in just two rooms that are adjacent, but I might be expanding into a dubstage which would then have a 60 foot run length.

Eric Bowen said:
Basically Fiber SFP+ copper has a far shorter distance limitation than Optical. Copper is cheaper though and requires lower power. The SFP+ 10Gbe adapters can have compatibility issues with certain SFP+ cables due to specs of the cable and the transmission built into the SFP+ adapters. If the 10Gbe network adapter is Intel then I suggest you get Intel SFP+ Twinax copper cables. Those adapters are very specific which cables they work with ie resistance and such. Ethernet overall though is cheaper and easier to deal with. 10Gbe Ethernet would be easier and cheaper to deal with if you have that option for the HP. I would suggest that since any Cat 6 or Cat 7 cable will work and distance options are better than copper SFP+.

http://www.cisco.com/c/en/us/produc...ransceiver-modules/data_sheet_c78-455693.html
Click to expand...
Thanks, the z840 is using the sfp+ while I use the ATTO boxes to the macs with cat 7. Long term i'll probably get more of the optical like you mentioned when/if I have a run out to a actual server room.

M.Halsell said:
Upgrading to 10 Gigbabit, just getting last feedback from the forum

@ Patrick F.: Any problems are hiccups with the Cisco sg500xg, happy with it?
@ Jeff K.: Any forthcoming switches that would preempt any of the current offerigs?
@ Anyone: For a Hackintosh build SmallTree vs ATTO vs Solarflare? Performance, price, stability?

Thanks in advance.

Sidebar: Have a Linksys Switch. I want to mod the fans to dampen the noise with a quieter fan such as Noctua. Doable or should I stay with stock fans for safety?
Click to expand...
My switch is still running strong, only thing is when I have 3 computers all totally banging at 10G the noise REALLY picks up. Right now I don't have a server closet, and depending on "how things go", i might really need to do that. I pretty much need to wear headphones when I do a render on one machine (two 10 core cpu's with k6000), a larger composite on my 12 core Mac Pro and trying to catch up on my other Mac Pro. The 200+ terabytes of disk though makes about the same amount of noise as the switch. My protools setup was also in this room, and I need to move that to another room(that will need to be 10g cat7).
 
Patrick, when the upper limits of the 10 gbe bandwith is being reached, how heavily are the CPU cores taxed?
 
M.Halsell said:
Patrick, when the upper limits of the 10 gbe bandwith is being reached, how heavily are the CPU cores taxed?
Click to expand...
I don't think the CPU's are ever waiting on the network, it's more like the network is waiting on disk. Perhaps 1 or 2 cores to read at 10g and send to at 10g(for short bursts it's reading from RAM and I doubt it's really even using a full core for that). I have eight Pegaguses2 32 gig raid drives, normally on run I'm mainly hitting 3 to 4 raids drives and those are often the limiter for the network. On rendering and/or compositing the network really is now where close to maxing out, since those are CPU and GPU bound by far. There are never more then 3 jobs running on the network though, and i'm the only one working on it ... so not a normal "shop" where a bunch of people are hitting it.
 
Here's some data from last year, in the form of a post on REDUSER.NET and ServerFault. The upshot is that on 2011-era Xeon processors, one CPU core is not adequate to reach 10Gbps but two are. And you may have to massage the operating system to get those two cores sufficiently isolated from the rest of the cores that they stay on task and everything else stays out of the way.

I suspect that even though we have more powerful processors, clock speeds have only gone up about 50% in the frequency-optimized realm, so two cores are still needed to do the 10Gbps job. But even if it's only one, you still need to keep the OS on the straight and narrow, partitioning the network-dedicated cores from the rest of the server's tasks.
 
You must log in or register to reply here.
Back
Top