#largefiles

[hbssacademy]

Parse and Analyze Large JSON Files | python

Discover efficient methods to parse and analyze large JSON files. Our comprehensive resources and tools simplify data handling for developer

[17thandoak]

Super large artwork files that take forever to save and impatience means having to do all my saving on the drive to my studio. I should of been a Web Designer. - #largefiles #forevertosave #drivetowork #batman #wip #comingsoon #firstworldproblems #gotham #illustrator #illustration #digitalart #design #designer #17thandoak

[ngodingkode-blog-blog]

How to remove largefiles from Mercurial repo

How to remove largefiles from Mercurial repo

See also this question.

Without knowing what I was doing, I enabled the largefiles extension, committed a file and pushed it to kiln. Now I know the error of my ways, and I need to permanently revert this change.

I followed the guidance from SO on the subject; and I can remove largefiles locally, but this doesn’t affect the remote repos in kiln. I have tried opening the repo in KilnRepositories…

View On WordPress

[sevennet-blog]

How to: Pushing Large Files to 500+ Computers #it #fix #dev

How to: Pushing Large Files to 500+ Computers #it #fix #dev

Pushing Large Files to 500+ Computers

I work with a team to manage 500-600 rented Windows 7 computers for an annual conference. We have a large amount of data that needs to be synced to these computers, up to 1 TiB. The computers are divided into rooms and connected through unmanaged gigabit switches. We prepare these computers ahead of time with the Windows installation and configuration, plus…

View On WordPress

[sevennet-blog]

Answer: RAID configuration for large NAS #programming #dev #answer

Answer: RAID configuration for large NAS #programming #dev #answer

RAID configuration for large NAS

I’m thinking of building a 24 1TB disk NAS box, but I’m not sure what the best drive configuration is. I’m looking at using the areca ARC-1280ML-2G controller, and hanging all 24 drives off of it.

I’d like it all to be mounted as one volume, due to the type of data we’re storing on it. One crazy idea we had was to configure 6 4-disk RAID 5 volumes, then do…

View On WordPress

[sevennet-blog]

Resolved: In Bash, are wildcard expansions guaranteed to be in order? #programming #it #dev

Resolved: In Bash, are wildcard expansions guaranteed to be in order? #programming #it #dev

In Bash, are wildcard expansions guaranteed to be in order?

Is the expansion of a wildcard in Bash guaranteed to be in alphabetical order? I am forced to split a large file into 10 Mb pieces so that they can be be accepted by my Mercurial repository.

So I was thinking I could use:

split -b 10485760 Big.file BigFilePiece.

and then in place of:

cat BigFile | bigFileProcessor

I could do:

cat…

View On WordPress

[sevennet-blog]

Resolved: Building a 1,000M row MySQL table #fix #development #it

Resolved: Building a 1,000M row MySQL table #fix #development #it

Building a 1,000M row MySQL table

This question is reposted from Stack Overflow based on a suggestion in the comments, apologies for the duplication.

Questions

Question 1: as the size of the database table gets larger how can I tune MySQL to increase the speed of the LOAD DATA INFILE call?

Question 2: would using a cluster of computers to load different csv files, improve the performance or kill…

View On WordPress

[sevennet-blog]

Solution: Which filesystem for large LVM of disks (8 TB)? #development #fix #answer

Solution: Which filesystem for large LVM of disks (8 TB)? #development #fix #answer

Which filesystem for large LVM of disks (8 TB)?

I have a Linux server with many 2 TB disks, all currently in a LVM resulting in about 10 TB of space. I use all this space on an ext4 partition, and currently have about 8,8 TB of data.

Problem is, I often get errors on my disks, and even if I replace (that is to say, I copy the old disk to a new one with dd then i put the new one in the server)…

View On WordPress