Coding in My Sleep @plottingourfictionallives - Tumblr Blog

The math that determines your whole life is racist & predatory, says Cathy O'Neil

O’Neil sees plenty of parallels between the usage of Big Data today and the predatory lending practices of the subprime crisis. In both cases, the effects are hard to track, even for insiders. Like the dark financial arts employed in the run up to the 2008 financial crisis, the Big Data algorithms that sort us into piles of “worthy” and “unworthy” are mostly opaque and unregulated, not to mention generated (and used) by large multinational firms with huge lobbying power to keep it that way. “The discriminatory and even predatory way in which algorithms are being used in everything from our school system to the criminal justice system is really a silent financial crisis,” says O’Neil.

The effects are just as pernicious. Using her deep technical understanding of modeling, she shows how the algorithms used to, say, rank teacher performance are based on exactly the sort of shallow and volatile type of data sets that informed those faulty mortgage models in the run up to 2008. Her work makes particularly disturbing points about how being on the wrong side of an algorithmic decision can snowball in incredibly destructive ways—a young black man, for example, who lives in an area targeted by crime fighting algorithms that add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation, which adds to a digital profile that could subsequently limit his credit, his job prospects, and so on. Yet neighborhoods more likely to commit white collar crime aren’t targeted in this way.

In higher education, the use of algorithmic models that rank colleges has led to an educational arms race where schools offer more and more merit rather than need based aid to students who’ll make their numbers (thus rankings) look better. At the same time, for-profit universities can troll for data on economically or socially vulnerable would be students and find their “pain points,” as a recruiting manual for one for-profit university, Vatterott, describes it, in any number of online questionnaires or surveys they may have unwittingly filled out. The schools can then use this info to funnel ads to welfare mothers, recently divorced and out of work people, those who’ve been incarcerated or even those who’ve suffered injury or a death in the family.

The usage of Big Data to inform all aspects of our lives, with and without our knowledge, matters not just because it dictates the life chances that are presented or denied to us. It also matters because the artificial intelligence systems that are being developed and deployed are learning from the data is collected. And those AI systems, themselves, can be biased and inaccessible to third-party audit.

Corporations are increasingly the substitutes for core state institutions. And as they collect and analyze data in bulk and hide away their methods of presenting data on behalf of states (or in lieu of past state institutions) the public is left vulnerable not just to corporate malice, but disinterest. Worse, this is a kind of disinterest that is difficult to challenge in the absence of laws compelling corporate transparency.

imathers

Chris (who I’m reblogging this from) is one of the smartest people I know, and since grad school he’s been the #1 person I’d listen to when it comes to technology, privacy, and the like. This is really, really crucial stuff.

plottingourfictionallives

Data is a weapon that has it’s own biases. It’s a weapon that can be used for both good and evil, but more often looks exactly like the people wielding it.

As someone who builds the processes surrounding big data (along with the data generation), accuracy isn’t the most important part. Who you are is represented by numbers, and updating the numbers on the small scale, individually, don’t affect the whole. Your numbers, what ‘your value is’ to individual corporations isn’t meaningful, even if it’s wrong. What the system thinks of you is what you are. Because, to marketing, internal organizations, you’re dollar signs, or you’re a target audience that isn’t yet tapped.

Just remember what Target is capable of when it has access to your shopping history? A lot. While in many ways it’s good. Because it reduces waste, it eliminates inefficiency, it also reduces you to a set of decisions that you make, without knowing scenario data. As @quirksintech points out, you don’t have access to this data. There’s a lot of data that Amazon, Google, Apple, Target, Wal-Mart keeps on people that you’ll just never have access to, that you can’t audit. It’s not like disputing an entry on your credit score, which would have been the most likely analogy ten years ago.

This is a larger, more damning issue when you apply this to quantitative ranking of people. When you eliminate relationships, and locality, and again, look at people purely as numbers. As a software engineer, I’ll readily admit that there’s a lot of scenarios in Big Data where we can’t properly engineer solutions to deal with context in a lot of situations, and thus, we just ‘guess’, and create coefficients that hope to reduce some variance in data in order to make decisions on it that exist within an acceptable margin of error here.

But, if you’re stuck in the margin of error, you’re in the dark, you don’t know why you are, and often you have no recourse when you are, and it’s not a good investment of time for corporations to change it, because they’ve already decided that you’re within the acceptable margin of error.

Have you ever wanted to use a particular CSS feature but didn’t because it wasn’t fully supported in all browsers?

“In 10 years you won’t have to wait 10 years anymore.”

#techniques #css #web development #browsers

bubble-lead

Important to remember for developers and pretty much everyone

npr

When people write computer programs, their biases can creep into code. Computer experts say it's important to talk about the problem because if humans don't fix it, computers won't do it themselves.

plottingourfictionallives

Listened to this last night on NPR.

I was talking to myself talking to the report, saying: Yes, of course it can be, but the issue is more in the raw data than the computer. Computers are merely representing what is reality here, and that is data is telling us that society is sexist. Society is racist. Society has biases, and while we use the data to make algorithms, if we’re making the machine learning work, it’s learning to ‘be more like us’. And that’s a bad thing.

Because we need impartial realities sometimes. Computer programs that use machine learning aren’t racist and sexist, but the way we mold them, iterate them, the algorithms that are applied to them, cannot correct for bias unless we explicitly code them as well.

#tech #racism #sexism

What maintainable software means? Part 1.

Over the last few months as the wave of feature requests slowly trickled to a halt for the first time in two years. We've suddenly been able to think about something important. How much did we hack onto our code in order to 'make it work'?

The answer. A lot. This isn't uncommon, and it's entirely symptematic of dealing with post-MVP growth stages in your application. You go through cycles of growth and refactor, growth and refactor. And we entered a time where our need to refactor grew greatly.

I was spearheading a lot of the conversations around this and what it means. Because, when you get to this level, the stasis between the two spots, you need to start questioning every single one of your design choices from before, especially in cases of rapid growth. Why did we build our data model like this, and why did we structure our routing like this, and what's the real best workflow like? After spending a bunch of time developing the front end and the data requirements at breakneck speed, we're forced to look at what we did wrong, why we did it, and what was the best course of action to take in the future, and how can we can prevent it?

Meetings aren't always a joyeous part of the job (although I tend to like these high value ones), but, we scheduled many meetings to clarify what we're trying to do in the context of our application. Our main goal was 'we want to build this as to open source it without any skeletons or black magic as to how it works', even if we never released it as open source, it was a goal to create transparency in the architecture and simplify and standardize how we handled certain situations. We spent nearly a month of our time documenting, transpiling the concepts, and creating stories in our backlog of how we wanted to handle these things. It's not an easy task by any stretch of imagination, but a clearly important one when you're making the shift from an MVP to a full fledged application that's maintainable.

We realized we had a maintanance problem, well... basically the entire time. Originally we had 3 microapps that were built around each other (developed independently) and a single SSO application to handle syncing between the them, and that poor design choice made it very difficult to maintain, so, we took the first step and merged all of the code together, and DRY'ed it all up. We still had significant debt. Like, the national debt sized debt for our team. When I say it was 'that large', I really do mean it. When your team is small, which ours is, it's imperative that you make sure you don't grow outside of what's capable to maintain. In the 'olden days' we used to use lines of code measurement as a rule of thumb of how many developers you need, but a lot of it has to do with having the right mix of developers across many levels to effectively manage and delegate application issues. Our team had a glut of senior level developers, but a lack of mid-level and junior-level meant a lot of the high level application development got blocked by tasks that were best suited for mid and junior level developers. It's typically a problem in smaller teams that that happens to roll up and accumulate over a period of time and significantly hurts team velocity and quality. In a perfect world, senior developers wouldn't just be coding, but actively taking time to ensure standards are enforced, to cross-check code, to read through and comment on pull requests and mentoring. Because when you're looking at maintainability, what you're really looking at is building consistency, where you don't have to explore the wild wild west every single time you step out into a new story. Where any developer can step in without arcane knowledge and do their job is the biggest key to maintainability.

So, after talking about how we've developed a strategy on what's wrong, the answer then becomes: how do we fix it. And for a lot of it, it was mostly reorganizing data, about following standards. As a Rails shop, that meant going by the book with some things, realizing that we had disorganized models and controllers that did too much, that we were stuffing too much logic into views. But we also had significant application design issues, such as how we handled our copious amounts of user data through duplication, normalization and denormalization. We didn't have a specific strategy for it. One of our strategies was to use some polymorphism, and carry similar, meaningful data around between our data models. We shared user scores and action events, progress notes and conversationable items as a flexible, yet restricted type of data in a consistant way. This means: when we looked at how our users were progressing in their action logs, we weren't having to check multiple models, we asked a single model to handle all of this. That clearly leads to consistancy, because when you look at how workflow is driven, you're not often handling a single model. It's the base flaw of using something like RailsAdmin right out of the box and expecting it to grow with your application. What's best for your data isn't best for your administrators, and not all of the time best for your application. It requires a gentle, invisible hand to start staying: what if we do this? What's the implications of that?

As we defined it, maintainablity meant we were able to minimize time spent on exisiting code. For us, that meant we needed to get a more robust test suite. I can't ever suggest this is absolutely critical in the quest to create maintainable code. People may quibble on if you need to run unit tests on static compiled languages like C# and Java, and every single one of those people are wrong. Someone you may know, someone you may like, someone who may be you, can write side effects in an application that have inadvertant consequences. You may end up spending more time validating that your code doesn't affect other code's results than actually writing the code. You may be spread out amongst a half dozen files, hunting things down. And yet, this is exactly why I say you need to build tests for everything. Being able to say: "tell me if this change anything turned red" is a massive tool in maintainability. You're not building code that breaks things unexpectedly. But again, you're only as good as your test code, and your test code leads to more maintainability. BUT THAT'S OKAY. When you're making the change, like we were, to more maintianable, more stable code, you're actually spending less hours in the end on each story. You can say: i'm spending X hours here, and there's then going to be Y hours saved in the end because this touches so many parts of the application, that we're going to be able to test quickly without having to spend a massive amount of time doing high touch testing. It's a waste of time, honestly, to do it any other way.

That, though, obviously requires us to build methods and objects that respect testing, that are small, don't rely on side effects, and return states. It's not a hard thing, but something that we found helps create better maintainable code as well. We took a hard look at documentation, and one of the things that we were lacking were clear directives of what each function returned, and what it was supposed to do. We prefered Yardoc in Ruby and Codo in Javascript in order to get a better handle of what each function was. To go back in time and say: well, I don't know exactly how this works, or why we did it like this, also is essentially an audit of your system. It's a way to say: if I don't understand this, how it's written, how will anyone else? And those are big maintainability smells.

Next week: Part 2 on Maintainability. What we actually did, what we didn't do, and how did it end up?

#maintainable software #coding #programming #rails #ruby #java #unit testing

boomeyer

Do yourself a favor. Learn to code. Here's how.

I’ve said this to my non-techie friends countless times. It’s no secret that being able to code makes you a better job applicant, and a better entrepreneur. Hell, one techie taught a homeless man to code and now that man is making his first mobile application.

Learning to code elevates your professional life, and makes you more knowledgeable about the massive changes taking place in the technology sector that are poised to have an immense influence on human life.

(note: yes I realize that 3/5 of those links were Google projects)

But most folks are intimidated by coding. And it does seem intimidating at first. But peel away the obscurity and the difficulty, and you start to learn that coding, at least at its basic level, is a very manageable, learnable skill.

There are a lot of resources out there to teach you. I’ve found a couple to be particularly successful. Here’s my list of resources for learning to code, sorted by difficulty:

Novice

Never written a line of code before? No worries. Just visit one of these fine resources and follow their high-level tutorials. You won’t get into the nitty-gritty, but don’t worry about it for now:

Dash - by General Assembly

CodeAcademy

w3 Tutorials (start at HTML on the left sidebar and work your way down)

Intermediate

Now that you’ve gone through a handful of basic tutorials, it’s time to learn the fundamentals of actual, real-life coding problems. I’ve found these resources to be solid:

Khan Academy

CodeAcademy - Ruby, Python, PHP

Difficult

If you’re here, you’re capable of building things. You know the primitives. You know the logic control statements. You’re ready to start making real stuff take shape. Here are some different types of resources to turn you from someone who knows how to code, into a full-fledged programmer.

Programming problems

Sometimes, the challenges in programming aren’t how to make a language do a task, but just how to do the task in general. Like how to find an item in a very large, sorted list, without checking each element. Here are some resources for those types of problems

Talentbuddy

TopCoder

Web Applications

If you learned Python, Django is an amazing platform for creating quick-and-easy web applications. I’d highly suggest the tutorial - it’s one of the best I’ve ever used, and you have a web app up and running in less than an hour.

Django Tutorial

I’ve never used Rails, but it’s a very popular and powerful framework for creating web applications using Ruby. I’d suggest going through their guide to start getting down-and-dirty with Rails development.

Rails Guide

If you know PHP, there’s an ocean of good stuff out there for you to learn how to make a full-fledged web application. Frameworks do a lot of work for you, and provide quick and easy guides to get up and running. I’d suggest the following:

Cake PHP Book

Symfony 2 - Get Started

Yii PHP - The Comprehensive Guide

Conclusion

If there’s one point I wanted to get across, it’s that it is easier than ever to learn to code. There are resources on every corner of the internet for potential programmers, and the benefits of learning even just the basics are monumental.

If you know of any additional, great resources that aren’t listed here, please feel free to tweet them to me @boomeyer.

Best of luck!

thetrekkiehasthephonebox

Oh perfect

plottingourfictionallives

Here’s some more resources for the fearless who, like me, want to pick up a book and learn how to do something:

Ruby on Rails

The industry standard introduction to Ruby on Rails

It’s basically what I tell every intern I have to read and display some knowledge of before they start. Hartl’s tutorial is extremely thorough and basically amazing.

Python

The way I learned to love tabbing the hard way.

Python the Hard Way is to Python what Hartl’s tutorial is to Ruby. It’s thorough, fun, and well written.

Lua

Your favorite language that treats the first in a series as 1 and not 0.

Lua is used a lot in the game development industry, it’s a bit different, but, easy to pick up and master because it’s a small language!

Haskell

You want functional programming? You don’t know what it is? HERE TAKE ANYWAY

Functional programming is the #nextbigthing. Saying you know it makes you look like a hot shot. It’s also a great way to learn the basics of programming because it’s clear, concise and won’t pollute your head with bad things.

If you have any questions, feel free to give me a ring. I’ll help you out day or night to point you in the right direction!

#programming #learn to program 2016

bubble-lead

At a SXSW panel yesterday called How Innovation Happens, Steve Jobs biographer Walter Isaacson, US Chief Technology Officer Megan Smith, and Google Executive Chairman Eric Schmidt discussed how an increased level of diversity and gender inclusivity is necessary to bring about breakthroughs in the tech sphere.

Dude……. Come on….

rubyloveinfo

How to Call Private Methods On Objects

rubyquicktips

Calling private methods can for example be useful in unit testing to increase the code coverage. Object#send gives you access to all methods of a particular object (even protected and private ones).

obj.send(:method [, args...])

In case send method has been overwritten, you can also use its aliased version __send__.

plottingourfictionallives

I always forget this, and always have to search for it.

And every single time I do, I say: Ruby’s private methods aren’t really ever private methods. Ever.

dothtml

Foundation for Apps, the first front-end framework created for developing fully responsive web apps.

rookiemag

Scene Report: Coding for Women’s Health

Where girls dominate computer programming!

By Maggie.

dothtml

Color Highlighter

plottingourfictionallives

this is sweet as hell.

Converting Salesforce Leads on Ruby

Salesforce and Ruby have a strange little twist together, considering that Heroku is owned by Salesforce and Heroku is a godsend for Rails developers everywhere for maintainable and manageable server environments. It's a strange pairing that I've learned about. Heroku maintains the fantastic databasedotcom Gem which is useful for querying the Salesforce REST API.

Originally we had a Salesforce Dev whip up something in Force.com to allow us to use the REST API to accept calls to natively convert Salesforce Leads. But somewhere along the lines, it was destroyed, overwritten by shoddy sandboxing and bad data, just 'disappeared' without a backup copy.

So, Sales kept pestering our Project Manager, asking to get this fixed, over, and over, and over again.

But, without having a Force.com developer. HOW? How would you do it? Easy: API THE SUCKER. Salesforce's API's are well documented, kinda crappy, but ultimately moldable and workable to do what you need them to do. Frankly, for the most part, moving data between a Ruby Application & Salesforce isn't even that hard! Heroku's making it even easier to do that, to move information from web portals to your Salesforce databases. But, converting a Lead? Couldn't be done without accessing the old SOAP API.

And thankfully, I have a lot of good SOAP experience working with old API's from other companies and have Savon stuffed in my back pocket for situations like this. So I went to fire up the good ol Bundler Gem Builder, and named it Soapforce, because, what else would you name a Gem that works with the Soap half of Salesforce?

And then I went to push the shell to Github, and there was already a Soapforce on Github, and granted it wasn't updated that recently, I was using a Rails 3.2.x application. So I vetted the gem, and said: well. This just took out about 6 hours that I would have been Gem Building. (cue Self Highfive). It uses Savon, and has a function called

def call_soap_api(method, message_hash={})

Perfect!

So, you'd authenticate, and then go

call_soap_api 'convert_lead', ...

From there, we checked out the SOAP documentation for the convert lead process.

Ah! The joy, the guts of SOAP. If you check your WSDL (or the sample Partner WSDL that Soapforce has in it's Library), you can find LeadConvert, which gives you the basic structure of your message_hash

Originally the Hash was set up with

{'accountId' => nil, 'contactId' => nil, 'convertedStatus' => 'Closed', 'doNotCreateOpportunity' => true}

And of course, it threw fun errors back at me like Element X invalid at this location because I wasn't mimicing the XML correctly, obviously.

{'LeadConvert' => {'accountId' => ...}}

Simple things that I wasn't expecting, that it wasn't expecting just a bunch of elements, but a sequence of items in XML.

With that though, the action actually became easy enough, we passed in our accountId and contactId that we made in the step before with data from our portal using databasedotcom, and then connected through Soapforce to our database, and bang boom, I was converting leads rather easily from then out.

I saw it posted about a few times on Stack Overflow without good direction on how to solve the issue, so I took it the next step and voila (or as some of my friends like to say: Whala!), you're now well connected with the SOAP half of Salesforce, and you get access to the following actions:

[login, describe_s_object, describe_s_objects, describe_global, describe_data_category_groups, describe_data_category_group_structures, describe_layout, describe_softphone_layout, describe_search_layouts, describe_search_scope_order, describe_tabs, create, update, upsert, merge, delete, undelete, empty_recycle_bin, retrieve, process, convert_lead, logout, invalidate_sessions, get_deleted, get_updated, query, query_all, query_more, search, get_server_timestamp, set_password, reset_password, get_user_info, send_email_message, send_email, perform_quick_actions, describe_quick_actions, describe_available_quick_actions]

So, for everything you don't actually have access to in the REST api, you definitely have access to in the SOAP side.

#SOAP #salesforce #ruby #ruby on rails #web development #api #rest

dothtml

plottingourfictionallives

By now I’m sure you’ve heard of Sass and how “You really need to start using it!” Learning a new tool can suck and finding the time to do so is next to

jerkcity-deactivated20180518

#5640: let’s play unix

azspot

Marketers and managers use jargon because it’s safe. No one stops them to ask: exactly what is it you are breaking through? What precisely are you transforming, and how are you certain the new thing will be better than the old (e.g. New Coke)? If no one, especially no one in power, challenges its use, jargon spreads, choking the life out of conversations and meetings forever. Pay attention to who uses the most jargon: it’s never the brightest. It’s those who want to be perceived as the best and the brightest, something they know they are not. They use cheap language tricks to intimidate, distract, and confuse, hoping to sneak past those afraid to ask what they really mean.

Why Jargon Feeds on Lazy Minds (via azspot)

See: all of IBM upper and middle management. You practically have to flay your own flesh in front of them to get a non-corporate-speak answer.

(via highsee)

i hear more than my fair share of this, everythings buzzword, not what you're capable of, or what you've done, but what people think you know. people who know how to leverage language to make it seem like they're in charge, from both a hiring and a hiree standpoint.

We're still digesting today's lengthy decision in the Oracle v. Google appeal, but we're disappointed—and worried. The heart of the appeal was whether Oracle can claim a copyright on Java APIs and, if so, whether Google infringed that copyright. According to the Federal Circuit today, the answer to both questions was a qualified yes—with the qualification being that Google may have a fair use defense.

this is also super bad.

rubywankenoobie-deactivated2020

:Symbols versus "strings" in Ruby

We had this question come up last night in BEWD. If you’ve ever seen a hash, you’ve probably seen this before:

states = {“New York” => “NY”, “California” => “CA”, “Florida” => “FL”}

or this:

states = {:New_York => “NY”, :California => “CA”, :Florida => “FL”}

Often the differences are explained as a matter of stylistic preference, but it goes beyond that. If you jump into IRB or Pry and create a variable, you’ll see the following:

I created a string and called object_id on it. What happens? It returns a set of numbers; if you look closely, each time I call object_id on “new york”, it returns a different number each time. What are those numbers? It’s where that string, “new york”, is saved in memory. One of a string’s properties is that it is mutable, or can be changed. The string I call is never the same string. It’s always different.

Now let’s observe what happens when I create a symbol:

I created a symbol, :new_york (run :new_york.class; you’ll see it’s a Symbol). When I call object_id on it, I also get back a set of numbers. The difference? Every time I do it, it returns the same number. It’s stored in the same place in memory which means a few things: 1. it saves memory (imagine how slow a program would be if a string was stored in a different place every time it was called; it), 2. it is immutable; it cannot be changed. The symbol represents the same value for the entirety of the program you’re running. Neat, huh?

Onward, noobs!

#ruby

Trending Blogs

Recently Viewed Blogs

Coding in My Sleep