#hdf5

[locyp]

Read some Data from a HDF5 file into memory. Get the dimensions, datatype and datasize.

[mktstk]

Big Data in Python: out of core processing

Recently we were presented with the opportunity to explore a big dataset in the Social Finance space. The size of the dataset was substantial: an order of magnitude greater than the core memory our computers. As a result, we couldn’t employ many of the usual methods for using Python’s Pandas with data that fits in memory. In fact, even if your data fits in core memory, if it is big enough you…

View On WordPress

[goodanswerbiz]

How to: "Large data" work flows using pandas #computers #fix #answer

How to: "Large data" work flows using pandas #computers #fix #answer

"Large data" work flows using pandas

I have tried to puzzle out an answer to this question for many months while learning pandas. I use SAS for my day-to-day work and it is great for it’s out-of-core support. However, SAS is horrible as a piece of software for numerous other reasons.

One day I hope to replace my use of SAS with python and pandas, but I currently lack an out-of-core workflow for…

View On WordPress

[sevennet-blog]

How to: "Large data" work flows using pandas

How to: "Large data" work flows using pandas

"Large data" work flows using pandas

I have tried to puzzle out an answer to this question for many months while learning pandas. I use SAS for my day-to-day work and it is great for it’s out-of-core support. However, SAS is horrible as a piece of software for numerous other reasons.

One day I hope to replace my use of SAS with python and pandas, but I currently lack an out-of-core workflow for…

View On WordPress