Bringing the funk to data. @bringingthefunktodata - Tumblr Blog

Why communication is important in Data science

It often happens that lower-level employees have long-established work habits that are required to change based on data-driven recommendations. Many times they are resistant to the change because they do not have enough context to understand why it is better that things change. That is why CLEAR communication is important.

It facilitates the flow of information. This compounds as the number of people working on a project grow.

It helps decision-makers act.

It removes misunderstandings, creates clarity, and motivates and unifies the participating teams.

It aids in preparing people to accept change.

It helps your manager coordinate with other managers and teams, so you can focus on the technical work.

All of this culminates in you making a greater impact on the organization

Things to consider:

1. What is the reason behind the request? : When someone makes a request, the outcome of that request is not usually the end goal, but rather a means to an end.

2. Once you know the reason behind the request, consider if the request is the right one. This often is not purely a data science task, but may also require domain knowledge (specific knowledge of the industry) or knowledge of operational constraints which you might not have

Why is the use of nebulous language ever so common?

People may not have the technical expertise to be more precise — not uncommon for managers and colleagues in non-technical teams (project managers and operational teams for instance).

People may not have the time to dig into the details — typical for management.

It's not other people's job to give concrete definitions to inherently vague terms

Proxies

It is a common situation in data science that when we don't know the truth — either because it's not possible to measure it or we can't afford to — we use sufficiently close approximations to the truth. These are called proxies. When a request is urgent, it is common to use proxies in place of information that would take much longer to get

API types (5)

GET

We use GET requests to retrieve information from a server (hence the name GET). With the GitHub API, we can use GET requests to retrieve user data.

e.g

response=requests.get(”API_URL”, headers=headers) actual_output=response.json( )

POST (201)

For example, we use POST requests to send information (instead of retrieving it) and to create objects on the API's server. With the GitHub API, we can use POST requests to create new repositories.

A successful POST request will usually return a 201 status code indicating that it was able to create the object on the server.

payload = {"name": "learning-about-apis"} response= requests.post("https://api.github.com/user/repos", json=payload, headers=headers) status= response.status_code

PATCH & PUT

We use PATCH requests when we want to change a few attributes of an object, but don't want to resend the entire object to the server. Maybe we just want to change the name of our repository, for example.

We use PUT requests to send the complete object we're revising as a replacement for the server's existing version.

payload= {"description": "Learning about requests!", "name": "learning-about-apis"} response=requests.patch("https://api.github.com/repos/VikParuchuri/learning-about-apis", json=payload, headers=headers) status=response.status_code

DELETE (204)

The final major request type is the DELETE request. The DELETE request removes objects from the server. We can use the DELETE request to remove repositories.

A successful DELETE request will usually return a 204 status code indicating that it successfully deleted the object.

Use DELETE requests carefully - it's very easy to remove something important by accident.

response=requests.delete("https://api.github.com/repos/VikParuchuri/learning-about-apis", headers=headers) status=response.status_code

APIs...finally

Here are some codes that are relevant to GET requests:

200 - Everything went okay, and the server returned a result (if any).

301 - The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint's name has changed.

401 - The server thinks you're not authenticated. This happens when you don't send the right credentials to access an API (we'll talk about this in a later mission).

400 - The server thinks you made a bad request. This can happen when you don't send the information the API requires to process your request, among other things.

403 - The resource you're trying to access is forbidden; you don't have the right permissions to see it.

404 - The server didn't find the resource you tried to access.

Connecting to a PostgreSQL database using psycopg2 is similar to connecting to a SQLite database using the sqlite3 library. psycopg2 also uses Connection and Cursor objects

Dataquest

Database engines: SqlLite Vs.PostgreSql

SQLite advantages:

The database is stored in a single file, making it portable.

You can use an SQLite database directly from Python and don't need a separate program running.

It implements most SQL commands, enabling you to use most of the statements you're familiar with.

SQLite disadvantages:

Only one process at a time can write to the database. When you have a complex web application, you may have multiple processes updating information in the database at the same time. For example, on Facebook, one process might handle updating user information, and another might handle generating the news feed.

You can't take advantage of performance features, such as caching (storing data in a temporary storage area). Because an SQLite database is a single file, and it doesn't require a special program to run, it can't have performance optimizations like caching. When running a site like Facebook that has a ton of traffic, it's important to be able to lookup data quickly.

SQLite doesn't have any built-in security. With a production website, it's common to want some people to be able to modify tables in a database (write), and others to only be able to make SELECT queries to tables in the database (read). This is because giving someone write access to the database can be a security risk, in that they can update or overwrite data. SQLite doesn't allow for restricting access to a database in this way.

<< That said, SQLite is good for small and simple database engine. SQLite is used extensively in embedded applications, such as Android and iOS applications >>

PostgreSQL is important for

multiple users or

When performance is important

At a high level, PostgreSQL consists of two pieces, a server and clients. The server is a program that manages databases and handles queries. Clients communicate back and forth to the server. Only the server ever directly accesses the databases -- the clients can only make requests to the server. The advantage of this model is that multiple clients can communicate with the server at the same time - allowing multiple processes to write to a database at the same time. By default, PostgreSQL uses a port 5432 to communicate with the outside world. If you start a PostgreSQL server, it will listen for incoming connections on port 5432

It can run remotely or locally

#data engines #data engineering #learn how to code #sqlite #postresql

And so today, I look at a couple of data visualization types

Downloading data sets

import numpy as np

import matplotlib.pyplot as pp

import seaborn

%matplotlib inline

import urllib.request

urllib.request.urlretriever('ftp.add_website_here', ‘add_name_of_file_here’)

OR...Download the file and skip all this to:

open(’file_name’, ‘r’).readlines ( )[:10]

Can’t keep calm....The first graph is here!!

import numpy as np

import matplotlib as pp

%matplotlib inline

x = np.linspace( 0,10,40)

sinx=np.sin(x)

cosx=np.cos(x)

pp.plot(x, sinx, ‘x’)

pp.plot(x, cosx, ‘o’)

#matplotlib #python #learn to code #learn python

Accessing files with python

The magic word is........

OPEN!!

Example:

// to open a file called words, you would do this by:

word = open (’words’, ‘r’)

// to print out everything that is in that document:

wordlist = word.readlines ( )

// type the first 10 lines of the document using the slice:

wordlist [:10]

//count the elements in the list

len(wordlist)

//use comprehension to strip words of funny characters and make them lowercase/uniform

wordclean = [word.strip( ). lower ( ) for word in wordlist)

//remove duplicates by turning the list into a python set ; a container that have 1 instance of any object then convert it back into the list

wordunique = list (set( wordclean) )

//then arrange it in alphabetical order:

wordunique.sort ( )

// And you can compress all the above by writing:

wordclean = sorted list (set ([ word.strip( ) .lower( ) for word in open(’words’, ‘r’) ]))

Standard variation measures the average amount by which variates are spread around the mean. To find out information about the population (such as mean and standard deviation), we do not need to look at all members of the population; we only need a sample. But when we take a sample, we lose some accuracy.

The range is simply the difference between the highest and lowest variate

The mode is the most frequently appearing variate or attribute in a population.

The median is another way of determining the "average" variate in the population. It is especially useful when the population has a particularly skewed frequency distribution. in a skewed distribution where a few large values fall into the high end or the low end of the scale, the median describes the typical or average variate more accurately than the arithmetic mean does.

The arithmetic mean is simply the average. It also has an interesting property: the sum of the deviations of the individual variates from their arithmetic mean is always is equal to zero

A proportion refers to a fraction of the population that possesses a certain property.

The aggregate is the sum of the values of all the variates in the population.

A parameter describes an entire population, whereas a statistic describes only a sample.

Trending Blogs

Recently Viewed Blogs

Bringing the funk to data.