
seen from Malaysia
seen from India
seen from Laos
seen from China
seen from Russia
seen from Singapore

seen from China
seen from China

seen from China
seen from United States

seen from United Kingdom
seen from China
seen from United States

seen from China
seen from China
seen from Australia

seen from China
seen from United States

seen from Canada
seen from Taiwan
Good idea: Publishing your data set as supplemental material to your paper.
Bad idea: Printing your complicated pivot table to PDF and publishing that as supplemental material.
Yes, I know some journals insist that supplemental material be PDF only. I know the authors probably did not have a choice in the matter.
But I am fighting for my life here trying to get these data into usable form.
Thank Christ for the developers of camelot-py, is all I can say.
Learning R and Reviewing Statistics
Happy new year everyone! This coming semester I will be taking my first SDS (statistical & data sciences) course at Smith. Even though I know very little about the department, SDS is my third choice major following my actual majors (engineering and computer science). To be totally honest, my main regret about my double STEM major has been the unofficial restriction it has placed on my exploration of SDS and other STEM fields. Fortunately, my engineering major requires that I take a statistics course! While most engineering students take SDS 220 (introduction to probability and statistics), I’ll be fulfilling the requirement by taking SDS 291 (multiple regression).
SDS 291 has a prerequisite of either SDS 201 (statistical methods for undergraduates), the aforementioned SDS 220, or ECO 220 (introduction to statistics and econometrics). A key difference between these introductory courses is in the prerequisites. Specifically, SDS 220 requires calculus I, and ECO 220 requires either microeconomics (ECO 150) or macroeconomics (ECO 153). While I technically have SDS 201 on my official transcript from having received a five on the AP statistics exam, I haven’t taken a statistics class in almost four years. (You can read more about my AP experience here). Since I knew my statistics knowledge was quite rusty, I emailed the professor to inquire about the specific concepts to review. In addition to recommending that I skim the first three chapters of a free online statistics textbook, I was advised to familiarize myself with R (a programming language used for statistical analysis).
Thus, I’ve spent the last two weeks (winter break) reviewing the basics of statistics and learning some R. I know that winter break is supposed to be a break from school and studying, but for the third consecutive year, I’ve made time for some independent learning. To be clear, I spent less than two hours studying each day, learning and programming are hobbies of mine, and independent learning is very different from homework and exams. During my first year, I reviewed computer science fundamentals in preparation for a placement test which you can read about here. During my sophomore year, I reviewed French in preparation for my interterm french course.
My statistics review began with me binge-watching Crash Course Statistics on double speed which got me thinking about the concepts before diving straight into the textbook. The specific textbook recommended by my professor was Introductory Statistics with Randomization and Simulation.
Next up was the fun part, learning a new programming language! I started out by reading the first four chapters of the second textbook recommended to me by my professor. The specific textbook was co-written by another Smith professor and is called Statistical Inference via Data Science A ModernDive into R and the Tidyverse. This text gave me a good introduction to using the tidyverse package to wrangle and visualize data.
To learn more, I turned to the various MOOC (massive open online course) platforms. I ultimately decided on Data Science: R Basics taught by a Harvard professor of biostatistics through edX. This course taught me about general programming in R and included interactive programming assignments to be completed through the DataCamp platform and on my local installation of R. I enjoyed the course and finished it faster than anticipated so I decided to take the next course in the series, Data Science: Visualization. This course more closely aligned with what I’d learned in the textbook but was still worthwhile thanks to the additional examples and graded exercises.
All in all, R is a fun programming language that I look forward to using and learning more about this spring. If you enjoyed this blog post, you may also enjoy reading about relearning Python, computer programming learning resources, and summer learning.
Fuck powerBI's UI, all my homies hate powerBI's UI.
https://missing.csail.mit.edu
The Missing Semester of Your CS Education
Best Data Science Course in Jalandhar
Advance your career with best Data Science course in jalandhar, offering practical skills and insights for success in the data-driven industry!"
https://techcadd.com/best-data-science-course-in-jalandhar.php/
Mastering Data Wrangling in SAS: Best Practices and Techniques
Data wrangling, also known as data cleaning or data preparation, is a crucial part of the data analysis process. It involves transforming raw data into a format that's structured and ready for analysis. While building models and drawing insights are important tasks, the quality of the analysis often depends on how well the data has been prepared beforehand.
For anyone working with SAS, having a good grasp of the tools available for data wrangling is essential. Whether you're working with missing values, changing variable formats, or restructuring datasets, SAS offers a variety of techniques that can make data wrangling more efficient and error-free. In this article, we’ll cover the key practices and techniques for mastering data wrangling in SAS.
1. What Is Data Wrangling in SAS?
Before we dive into the techniques, it’s important to understand the role of data wrangling. Essentially, data wrangling is the process of cleaning, restructuring, and enriching raw data to prepare it for analysis. Datasets are often messy, incomplete, or inconsistent, so the task of wrangling them into a clean, usable format is essential for accurate analysis.
In SAS, you’ll use several tools for data wrangling. DATA steps, PROC SQL, and various procedures like PROC SORT and PROC TRANSPOSE are some of the most important tools for cleaning and structuring data effectively.
2. Key SAS Procedures for Data Wrangling
SAS offers several powerful tools to manipulate and clean data. Here are some of the most commonly used procedures:
- PROC SORT: Sorting is usually one of the first steps in data wrangling. This procedure organizes your dataset based on one or more variables. Sorting is especially useful when preparing to merge datasets or remove duplicates.
- PROC TRANSPOSE: This procedure reshapes your data by converting rows into columns or vice versa. It's particularly helpful when you have data in a "wide" format that you need to convert into a "long" format or vice versa.
- PROC SQL: PROC SQL enables you to write SQL queries directly within SAS, making it easier to filter, join, and aggregate data. It’s a great tool for working with large datasets and performing complex data wrangling tasks.
- DATA Step: The DATA step is the heart of SAS programming. It’s a versatile tool that allows you to perform a wide range of data wrangling operations, such as creating new variables, filtering data, merging datasets, and applying advanced transformations.
3. Handling Missing Data
Dealing with missing data is one of the most important aspects of data wrangling. Missing values can skew your analysis or lead to inaccurate results, so it’s crucial to address them before proceeding with deeper analysis.
There are several ways to manage missing data:
- Identifying Missing Values: In SAS, missing values can be detected using functions such as NMISS() for numeric data and CMISS() for character data. Identifying missing data early helps you decide how to handle it appropriately.
- Replacing Missing Values: In some cases, missing values can be replaced with estimates, such as the mean or median. This approach helps preserve the size of the dataset, but it should be used cautiously to avoid introducing bias.
- Deleting Missing Data: If missing data is not significant or only affects a small portion of the dataset, you might choose to remove rows containing missing values. This method is simple, but it can lead to data loss if not handled carefully.
4. Transforming Data for Better Analysis
Data transformation is another essential part of the wrangling process. It involves converting or modifying variables so they are better suited for analysis. Here are some common transformation techniques:
- Recoding Variables: Sometimes, you might want to recode variables into more meaningful categories. For instance, you could group continuous data into categories like low, medium, or high, depending on the values.
- Standardization or Normalization: When preparing data for machine learning or certain statistical analyses, it might be necessary to standardize or normalize variables. Standardizing ensures that all variables are on a similar scale, preventing those with larger ranges from disproportionately affecting the analysis.
- Handling Outliers: Outliers are extreme values that can skew analysis results. Identifying and addressing outliers is crucial. Depending on the nature of the outliers, you might choose to remove or transform them to reduce their impact.
5. Automating Tasks with SAS Macros
When working with large datasets or repetitive tasks, SAS macros can help automate the wrangling process. By using macros, you can write reusable code that performs the same transformations or checks on multiple datasets. Macros save time, reduce errors, and improve the consistency of your data wrangling.
For example, if you need to apply the same set of cleaning steps to multiple datasets, you can create a macro to perform those actions automatically, ensuring efficiency and uniformity across your work.
6. Working Efficiently with Large Datasets
As the size of datasets increases, the process of wrangling data can become slower and more resource-intensive. SAS provides several techniques to handle large datasets more efficiently:
- Indexing: One way to speed up data manipulation in large datasets is by creating indexes on frequently used variables. Indexes allow SAS to quickly locate and access specific records, which improves performance when working with large datasets.
- Optimizing Data Steps: Minimizing the number of iterations in your DATA steps is also crucial for efficiency. For example, combining multiple operations into a single DATA step reduces unnecessary reads and writes to disk.
7. Best Practices and Pitfalls to Avoid
When wrangling data, it’s easy to make mistakes that can derail the process. Here are some best practices and common pitfalls to watch out for:
- Check Data Types: Make sure your variables are the correct data type (numeric or character) before performing transformations. Inconsistent data types can lead to errors or inaccurate results.
- Be Cautious with Deleting Data: When removing missing values or outliers, always double-check that the data you're removing won’t significantly affect your analysis. It's important to understand the context of the missing data before deciding to delete it.
- Regularly Review Intermediate Results: Debugging is a key part of the wrangling process. As you apply transformations or filter data, regularly review your results to make sure everything is working as expected. This step can help catch errors early on and save time in the long run.
Conclusion
Mastering data wrangling in SAS is an essential skill for any data analyst or scientist. By taking advantage of SAS’s powerful tools like PROC SORT, PROC TRANSPOSE, PROC SQL, and the DATA step, you can clean, transform, and reshape your data to ensure it's ready for analysis.
Following best practices for managing missing data, transforming variables, and optimizing for large datasets will make the wrangling process more efficient and lead to more accurate results. For those who are new to SAS or want to improve their data wrangling skills, enrolling in a SAS programming tutorial or taking a SAS programming full course can help you gain the knowledge and confidence to excel in this area. With the right approach, SAS can help you prepare high-quality, well-structured data for any analysis.
So excited to write again! Hopefully my homework doesn’t completely delete itself and sabotage my schedule