No title available
Not today Justin
hello vonnie
Claire Keane
todays bird
$LAYYYTER
Mike Driver
Cosmic Funnies
Monterey Bay Aquarium
"I'm Dorothy Gale from Kansas"
DEAR READER
★
KIROKAZE
macklin celebrini has autism
blake kathryn
tumblr dot com
Jules of Nature
Peter Solarz
RMH
occasionally subtle
seen from United Kingdom
seen from Germany
seen from Germany
seen from Netherlands
seen from China
seen from Türkiye
seen from United Kingdom
seen from Germany
seen from Malaysia
seen from United States
seen from Germany
seen from Netherlands
seen from Uruguay
seen from Malaysia
seen from Netherlands
seen from United States
seen from Germany
seen from United States
seen from United States
seen from United States
@hatarilabs-blog
Tutorial to crawl the latest post with Python Scrapy
Now Hatariwater is Hatarilabs!
Please visit our site at: www.hatarilabs.com
Quick tutorial to do a web crawl for the latest post with Python Scrapy
View files on the github repository: https://github.com/SaulMontoya/Tutorial-to-crawl-the-latest-post-with-Python-Scrapy
How to set humidity sensors in the "cloud"?
Now Hatariwater is Hatarilabs!
Please visit our site at:
www.hatarilabs.com
The technology is closer to water resources. It is now possible to measure physical parameters of the water cycle, put them in the "cloud" and access in real time. We talk about open source technologies.
This article shows the implementation of soil moisture sensors for periodic data collection, storage and online data representation on the web.
Tools
This project used open source software and open source hardware such as:
Soil moisture sensor: It measures the volumetric water content in the soil through its electrical resistance.
Arduino UNO: Microcontroller for the construction of digital gadgets and interactive objects. It was used to take soil hygrometer data and send the digital data to the Raspberry Pi.
Raspberry Pi: Programmable computer with digital ports, WiFi and USB connection. It was used to receive data from the Arduino board and send them to the cloud.
Python: Programming language used in the Raspberry Pi to connect to the Arduino and send the data to the cloud through ThingSpeak
Thinkspeak.com: Online platform for the "Internet of Things". It allows free online data storage to a certain amount. There are other platforms as nimbits.com free initial access, plot.ly, zettajs.org, among others.
About prices
This is a very interesting topic when we use open source hardware. The prices of these devices are very low compared to those distributed by commercial firms.
From Ebay prices with free shipping to anywhere in the world, these devices cost in USD Dollars:
Raspberry Pi 2: $ 38
Arduino UNO: $ 3.5
Humidity Sensor: $ 1.0
With power supply connector wifi, among other memories online price is $ 70 sensor capable of measuring up to 6 points of moisture.
Code
You can access the code in our GitHub repository:
https://github.com/SaulMontoya/How-to-set-humidity-sensors-in-the-cloud
How to smart search through files with Python
Now Hatariwater is Hatarilabs!
Please visit our site at: www.hatarilabs.com
Quick tutorial about smart search of parts of a file located in many directory. Exercise done with Jupyter Notebook
Download all the example data and the Ipython Notebook from our GitHub repository:
https://github.com/SaulMontoya/How-to-smart-search-through-files-with-Python
How to apply styles to rasters in GIS?
Tutorial about how to apply styles in QGIS.
Download the data from this exercise here:
AsterDEM
WaterTable
Now Hatariwater is Hatarilabs!
Please visit our site www.hatarilabs.com
Tutorial about how to apply styles in QGIS. You can see the video in our blog and download the files in this URL: http://how-to-apply-styles-to-rasters-in-gis
You can allways visit our blog: https://www.hatarilabs.com/ih-en/
Follow us on: facebook.com/hatarilabs twitter.com/hatarilabs
How to make banners for social networks with Python
Now Hatariwater is Hatarilabs!
Please visit our site www.hatarilabs.com
Sometimes we want to make a banner to share something on the social networks. Banners are more likely to be "liked" and shared rather than sharing the link alone. We can make a banner like this using software like Inkscape, GIMP, Office or other. We got 22 likes and 5 shares on Facebook.
You can access the hole the whole code on our Github profile:
https://github.com/SaulMontoya/how-to-made-banners-with-Python
In[]:
from IPython.display import Image Image(filename='Png/Options/bannerexample.jpg')
Out[]:
However, in order to do a banner for every idea, new content or event will become very tedious. If you are not a wealthy startup or a part of a organization with graphic designers you must to spend hours in becoming more success on the social networks.
In the beginning, Python was not enough
We wanted to make the banner just with pure Python code by using matplotlib. We had some success at the beginning, but finally we crash on putting the logo on the canvas. It was quite frustrating to spend hours of programming and then drop the code because is useless. But we didn’t gave up. The idea of having this tool was still of our interest and we came back to Python but with a little modification on the scope. That means that we were going to use Python to manage Latex.
How does the Python - Latex code work
The code does basically the following:
Open the image
Puts a background
Set the phrase / text
Adds a logo
Export as PDF
Convert the PDF into JPG
In[]:
# This is the list of packages we will need %pylab inline from PIL import Image from shapely.geometry.polygon import Polygon from descartes import PolygonPatch from matplotlib import pyplot as plt from matplotlib.pyplot import imshow
In[]:
#We define the function that set ups the backgroun def BackgroundGenerator(imagen): #---------- #Definimos poligono poligonotexto = Polygon([(0, 200), (1200, 200), (1200, 700), (0, 700)]) polycolor = PolygonPatch(poligonotexto, color='#3F83B7',alpha=0.7) #---------- #Definimos figura y subplot fig = plt.figure(figsize=(12,9), dpi=100) ax = fig.add_subplot(111) ax = fig.add_axes([0,0,1,1]) ax.add_patch(polycolor) #---------- #Definimos la imagen principal #quitamos el borde de la principal ax.spines['top'].set_visible(False) ax.spines['right'].set_visible(False) ax.spines['bottom'].set_visible(False) ax.spines['left'].set_visible(False) plt.tick_params( axis='both', which='both', bottom='off', top='off', right='off', left='off', labelleft='off', labelbottom='off') ax.imshow(imagen) return fig
# the code also resizes the image to 1200 x 900 otherwise the banner aligment can be screwed from PIL import Image img = Image.open ('Png/Options/rawimage.png') resize = img.resize((1200,900)) fig = BackgroundGenerator(resize) fig.savefig('Png/Input/imagen.png')
Out[]:
#This code puts the text and logo on the figure import re import os
keywords = ['FRASE1','FRASE2', 'FRASE3', 'FRASE4','IMAGEN'] datos = ['Seminario 13 Febrero', 'Modelamiento Numérico de Bofedales', 'y Manantiales y Evaluación del Impacto', 'de la Minería','gidahatari.png']
bannerentrada = open("Tex/HatariLabs_Lema.tex") bannerentrada = bannerentrada.read()
bannersalida = open("Tex/HatariLabs_Lema_Salida.tex", "w")
for i in range(len(keywords)): bannerentrada = re.sub(keywords[i], datos[i], bannerentrada)
bannersalida.write(bannerentrada) bannersalida.close()
os.system("pdflatex " + "-interaction=nonstopmode "+ 'Tex/HatariLabs_Lema_Salida.tex')
We have create a PDF of our banner. However, we can not share PDF as images on Facebook therefore we need a last code to convert the PDF to PNG
os.system("convert -density 100 -trim " + "HatariLabs_Lema_Salida.pdf " + '-quality 100 ' + 'Png/Output/HatariLabs.png') from IPython.display import Image Image(filename='Png/Output/HatariLabs.png')
How to effective manage tons of emails on Gmail with Python
Now Hatariwater is Hatarilabs!
Please visit our site www.hatarilabs.com
For a website we have (gidahatari.com) there is a form to sign up for a newsletter. Since we use a no-sql hosting provider as Squarespace there was not option to deploy a mailing software as phplist or similar.
Suddenly, every day we had more than 30 mails for the newsletter register. At the beginning a collaborator could manage to put them manually in the register, but after a couple of weeks the work was tedious and not motivating.
We got to the idea that this was a problem that our superhero Python could manage with a dozen of lines of code.
Notebook repository
For this script we have used Ipython Notebook. If you want to see the whole script you can visit the Ipython link on Github.
1. Open the email through Gmail Imap
In []:
# import the required libraries
import email, getpass, imaplib, os
# connecting to the gmail imap server
m = imaplib.IMAP4_SSL("imap.gmail.com")
m.login('[email protected]','yourpassword')
m.select("INBOX") # opens the inbox and gives the total amount of emails
Out[]:
('OK', ['4'])
2. Open the all emails and get the mail id
In []:
#searching for all email with imap rules resp, items = m.search(None, "ALL") # looking for mails id items = items[0].split()
3. Searching for emails with a particular content
Here is where magic comes true. All the email for the newsletter subscription have a regular content. Therefore we can search on the docens of email looking for the emails that fit the newsletter subcription content.
In this case we have set that all newsletter subscription emails stars on line 5 with the expression ‘Corre’
In []:
a = [] for emailid in items: resp, data = m.fetch(emailid, "(BODY[TEXT])") email_body = data[0][1] # getting the mail content mail = email.message_from_string(email_body) for linea in email_body.split('\n'): if linea[:5]=='Corre': a.append(linea[26:-1]) a
Out []:
4. Saving results on a text file
In []:
text_file = open("Output.txt", "w") for item in a: text_file.write(item + '\n') text_file.close()
5. Purging all repeated emails
Some persons send emails more than one time. Therefore we have to purge all duplicated emails.
In []:
archivo = 'Output.txt' contactos = open(archivo).read().splitlines() contactospurgados = set(contactos) # print all purged email and total emails print len(contactospurgados), len(contactos)
Out []:
4 4
In []:
text_file = open(archivo, "w") for item in contactospurgados: text_file.write(item + '\n') text_file.close()
Statistical analysis of precipitation data with Python
Now Hatariwater is Hatarilabs!
Please visit our site www.hatarilabs.com
Usually we use probabilistic approaches when dealing with extreme events since the size of available data is scarce to address the maximum for a determined return period.
Precipitation data present challenges when we try to fit to a statistical distribution. This is an exercise done with 10 years record of precipitation for the Morropon station (Piura-Peru) affected by El Niño and La Niña events in years 98 and 99 respectively (download data here).
Data processing was done with Python on the IPython Notebook interface. You can use the IPython cloud version in wakari.io to perform this exercise.
1. Import data
First we import data and plot it using panda library.
In[]:%pylab inline %cd C:\WRDAPP\Curso_PyH\AuxData import pandas as pd import numpy as np datos = pd.read_csv('PptMorropon.csv', index_col=0, na_values=[-9999], parse_dates=True) datos.plot() Out[]:
2. Create a histogram of precipitation
We create an histogram of the values of precipitation, meaning values higher than 0.
In[]:datos[datos>0].hist() Out[]:
As we can see, precipitation data are quite skewed. We will work on the distribution to fit it with "acceptably uncertainty" to a statistical distribution.
3. Plot of the normal and lognormal distribution
We calculate the mean and the standard deviation
In[]:promedio = datos[datos>0].mean() desviacion = datos[datos>0].std()
Then, we calculate the correspondent statistical distribution
In[]:tabulaciones=np.arange(-40,51,0.1) distnormal = stats.norm.pdf(tabulaciones, loc=promedio, scale=desviacion) distlognormal = stats.pearson3.pdf(tabulaciones,skew=1,loc=promedio, scale=desviacion)
And we plot the data
In[]:datos[datos>0].hist(bins=100,normed=True) plot(tabulaciones,distnormal, color='red', label='distnormal') plot(tabulaciones,distlognormal, color='grey', label='distlognormal') xlim(0,30) legend(loc='upper right')
Since the adjustment of the normal and lognormal distribution depend on the arithmetic mean and the standard deviation, those are measure locations not conservative and are highly affected by the extreme events. We have to "shape" the data in order to have a precipitation that fits reasonably a statistical distribution.
4. Trimming data
We decided to cut the lower 10 percent and the upper 20 percent of the values
In[]:print datos_ppt.quantile(0.10),datos_ppt.quantile(0.80) Out[]: Ppt_mm 0.4 dtype: float64 Ppt_mm 8.44 dtype: float64
We cut the data using obtained quantiles
In[]:datos_ppt_min=datos_ppt[datos_ppt>0.4] datos_cut=datos_ppt_min[datos_ppt_min<8.44] datos_cut.describe() Out[]: Ppt_mm count 295.000000 mean 2.383729 std 2.029009 min 0.400000 25% 0.800000 50% 1.600000 75% 3.500000 max 8.400000
5. Plotting trimmed data
We define new statistics for the trimmed data and plot statistical distributions.
In[]: promedio_cut=datos_cut.mean() desviacion_cut=datos_cut.std() tabulaciones_cut=np.arange(-10,20,0.1) distnormal_cut = stats.norm.pdf(tabulaciones_cut, loc=promedio_cut, scale=desviacion_cut) distlognormal_cut = stats.pearson3.pdf(tabulaciones_cut,skew=1,loc=promedio_cut, scale=desviacion_cut) datos_cut.hist(normed=True,bins=15) plot(tabulaciones_cut,distnormal_cut, color='red', label='distnormal') plot(tabulaciones_cut,distlognormal_cut, color='grey', label='distlognormal') xlim(0,15) legend(loc='upper right') Out[]:
Now we have two statistical distributions that fit better to the precipitation. We can say that lognormal distribution fits better to the recorded data.
Download the data and the ipynb on this link in Github.
Statistical methods in water resources with Python
Now Hatariwater is Hatarilabs!
Please visit our site www.hatarilabs.com
The following text deals with statistics on water resources coupled with related Python codes. The theory comes from Chapter 1 of Statistical Methods in Techniques of Water Resources Investigations, Helsel (2002). Scripts are written in Python with the IPython interphase.
Measures of Location
Classical Measure - the Mean
The mean (X) is computed as the sum of all data values Xi, divided by the sample size. Data points further from the center exert a stronger downward force than those closer to the center. If one point near the center was removed, the balance point would only need a small adjustment to keep the data set in balance. But, if one outlying value was removed, the balance point would shift dramatically. Mean is not a "resistant" measure of location.
Figure 1. Mean dependancy from extreme values from Helsel (2002).
Resistant Measure - the Median
The median, or 50th percentile P0.50, is the central value of the distribution when data are ranked in order of magnitude. The median is only minimally affected by the magnitude of a single observation. This resistance to the effect of a change in value or presence of outlying observations is often a desirable property.
Exercise:
In[]: import numpy as np a = np.array([2,4,8,9,11,11,12]) b = np.array([2,4,8,9,11,11,120]) print "The arithmetic median of the set a is:", np.mean(a) print "The arithmetic median of the set b is:", np.mean(b) print "But" print "The mean value of the set a is:", np.median(a) print " The mean value of the set b is:", np.median(b) Out[]: The arithmetic median of the set a is: 8.14285714286 The arithmetic median of the set b is:23.5714285714 But The mean value of the set a is:9.0 The arithmetic median of the set b is:9.0
Other Measures of Location
Three other measures of location are less frequently used: the mode, the geometric mean, harmonic mean and the trimmed mean.
Mode
It is the value having the highest bar in a histogram. It is far more applicable for grouped data, which are recorded only as falling into a finite number of categories, than for continuous data. It is very easy to obtain, but a poor measure of location for continuous data, as its value often depends on the arbitrary grouping of those data.
Exercise:
In[]: import numpy as np import scipy.stats a = np.array([1,2,3,1,2,1,1,1,3,2,2,1]) b = (3,4,5,6,7,8,5,5,6,2,3,1,2,5,5,1,3,2,2,1) print "The mode of the set a is: ", scipy.stats.mode(a) print "The mode of the set b is:", scipy.stats.mode(b) #if we do not like to see the parentheses we should use the following script print "The mode of the set a is:", scipy.stats.mode(a)[0][0], "and it is repeted",scipy.stats.mode(a)[1][0] ," times" print " The mode of the set a is:", scipy.stats.mode(b)[0][0], "and it is repeted",scipy.stats.mode(a)[1][0], " times " Out[]: The mode of the set a is:(array([ 1.]), array([ 6.])) The mode of the set b is: (array([ 5.]), array([ 5.])) The mode of the set a is: 1.0 and it is repeted 6.0 times The mode of the set b is: 5.0 and it is repeted 6.0 times
References
Helsel, D.R. and R. M. Hirsch, 2002. Statistical Methods in Water Resources Techniques of Water Resources Investigations, Book 4, chapter A3. U.S. Geological Survey. 522 pages.
