Web Semantics and Components

Pretty much anything

[defaultnamehere]

Graphing when your Facebook friends are awake

Look I’m not really sure why but I think I made a thing that makes graphs of when people are online on Facebook. It sounds kinda creepy and uh it is. Read along so you, too, can be the NSA. ˙ ͜ʟ˙

Little green dots

You know those green dots on the sidebar on Facebook that tell you who’s online? How do they get there?  Also there are times next to people who are offline. What are those about?

I was wondering the same things, and so one day I decided to 360 noscope hack Facebook by right clicking and selecting “Inspect Element”.

I’M IN

We did it team. Anyway alright uhhhh let’s just uh snoop around here reallllll sneaky like

If you reload the page you’ll see approximately fifty-bajillion network requests go off as Facebook desperately tries to load all the junk that it needs to display facebook.com. 

You might be wondering at this point why I decided to look for interesting things in this mess instead of, I dunno, getting out more, getting a cat, that sorta thing. Anyway hey look a heading

Finding the good stuff

What’s this “pull” thing?

THAT looks like some #datascience right there. This is the kind of 100% legit secret undocumented “API” that we came here for. Let’s do some reverse-engineering.

It looks like a mapping of Facebook user ids to… their online status? But there’s more than one value? “webStatus” and “fbAppStatus” are both there. What’s more, it tells you what the person is doing on each of the different kinds of statuses. 

For example:

“messengerStatus” : “invisible” means they’re not online on the Facebook Messenger app. 

“webStatus”: “idle” means their web browser is logged in to Facebook, and has the page open, but they aren’t doing anything on the site like moving their mouse or talking to anyone.

Since we have both of these at the same time, we can tell that this person is likely not using their phone, and that they were using facebook.com recently, but not right now.

That’s already a little creepy that we can tell that about people. But can we do more with this?

You might also notice that there is a value called “la” that is a big integer that starts with “14″. If you I dunno, didn’t have a lot of friends in high school, you might recognise that as a UNIX time stamp - the time in seconds since midnight, January 1, 1970. 

Computer Scientists thought this would be a good time to start measuring the time from because the first app was born at midnight, January 1, 1970. The app was a custom emoji pack for an ancient model of phone that would one day evolve to become the first Blackberry.

If you’re wondering why the response starts with “for (;;);”, it’s to, among other things, encourage developers to use a quality JSON decoder, instead of like, y’know, eval().

Anyway that “la” thing stands for “last active”, and tells you the last time the person was active on Facebook, down to the second. Do you see where I’m going with this?

Roleplaying as the NSA ˙ ͜ʟ˙

So far we have a whole bunch of things which look like this

A person

A time

Whether they’re online or offline or idle

Which devices they’re online/offline/idle on

This doesn’t seem that interesting at first, since you already know who is online by looking at the sidebar. But what if there was someone always watching the little green dots?

Using the power of computers, you can just write a Python program to listen to what the /pull requests are saying all the time ever, and write it down.

Here’s a screenshot of all the log files I’ve got:

And here’s what an individual log file looks like (the first 10 lines):

Those blurred out things are Facebook user ids. If you think these screenshots look a little bit creepy then YEAH I KNOW RIGHT. 

Tell me about your program then you massive nerd

It runs 24/7, and it’s constantly logging online/offline activity data from those /pull URLs using my Facebook cookie.

Writing it was mostly about saying “jeez, all these parameters look complicated” and then blindly copy/pasting them anyway.

Protip, you can right click on any network request in Chrome’s Developer Tools and click “Copy as cURL”. This is amazing and lets you re-run a request from the terminal, as well as give you all the headers and cookies used to run that request in a nice copy-pasteable format.

The first step was to just run that request verbatim in a terminal with curl. 

curl ‘https://1-edge-chat.facebook.com/pull?channel=p_751783962&seq=3&partition=-2&clientid=[redacted]&cb=6dcn&idle=5&qp=y&cap=8&tur=1545&qpmade=1455427171900&pws=fresh&isq=221841&msgs_recv=3&uid=[redacted]&viewer_uid=[redacted]&sticky_token=239&sticky_pool=atn2c06_chat-proxy&state=active’ -H 'origin: https://www.facebook.com’ -H 'dnt: 1’ -H 'accept-encoding: gzip, deflate, sdch’ -H 'accept-language: en-US,en;q=0.8,en-AU;q=0.6’ -H 'user-agent: ‘[redacted]’ -H 'accept: */*’ -H 'referer: https://www.facebook.com/’ -H 'authority: 1-edge-chat.facebook.com’ -H 'cookie: ‘[redacted]’ –compressed

I was expecting it to not work because it looks like it has some sequence numbers in it oh boy BUT it turned out to just take a really long time. I later found out this was because the /pull endpoint is using HTTP Long Polling, which turns out to be like a streaming HTTP GET request.

The only other important parameter to worry about is “seq”, which I’m guessing is the sequence number of the response from Facebook. Just add 1 to the sequence number that the response from /pull gives for the next request and you’re good to go.

If you’re worrying about remembering all this, chill out I got yo’ back, my 100% Terms of Service Compliant implementation of this is available here on GitHub. Standard disclaimers of “I’m so sorry I wrote parts of this in like 30 minutes” apply.

One caveat of the data-collection program that I’ve noticed is that it has false negatives. That is, sometimes it won’t give you a “this person is online” data point, even though they really are online. I guess that gives plausible deniability of… being offline? 

You should probably get out more

[worried laughter]

So that’s the hard part done, right?

Let me paint you a word-picture. It’s 11pm, I’m listening to the soundtrack to The Social Network (ironically? meta-ironically? I don’t even know), I have six terminals tiled across two screens as well as fifty thousand browser tabs open and I’m up to my third graphing library.

Making graphs is really hard.

I used matplotlib, but I realised this wasn’t my thesis and I wouldn’t be embedding this ugly graph as a pdf into a LaTeX document that takes 3 passes of pdflatex to render because there’s been a terrible but extremely localised accident where only humanity’s LaTeX to pdf converters have been irreversibly sent back in time to the 80s.

I used bokeh, which claims to be a “matplotlib-killer”, and it was was okay until a friend told me “it isn’t the 90s anymore, you don’t generate graphs server-side. Also your graphs are ugly and you should feel ugly you utter fraud”.

This friend recommended nvd3.js, presumably because you’re not making real graphs in 2016 unless your graphing library is <something>.js and requires at LEAST one other <something else>.js as a dependency. Everyone looks at you like “what, you DON’T already use <something else>.js? Jeez say goodbye to your Hacker News karma. Just apt-get install npm && npm install bower && bower install-” NO STOP IT THIS ISN’T WHAT TIM BERNERS-LEE WANTED”.

I think it took about three times as much time to graph the data as it took to write the code to download it. And the graphs aren’t even good! I gave up on perfecting the graphs so I could just hurry up and write this questionable blog post already. Just think of me resolving pip3 dependencies when you see the ugly graphs.

(°ロ°)☝ AND ANOTHER THING when it’s midnight and your x-axis formatting function doesn’t convert UNIX times into JavaScript date objects properly because there’s no timezone information and I dunno JavaScript was written by some guy in two weeks (yeah I ain’t afraid to call it out what of it) and your binary-search based conversion of sparse timeseries data into uniformly dense timeseries data is causing so many data points to be graphed that it’s slowly crashing Chrome and you’re watching helplessly as your RAM goes up and Chrome won’t close the tab and it just doesn’t seem right that 2016, the year of the Linux Desktop has brought us this situation I mean I thought if you had enough <something>.js libraries this stuff was meant to just scale right up so tha-

Quit stalling with graphing libraries and show me the graphs

Fine but you’re missing out on top-quality graphing-related banter.

The graphs in this section are all of the online/offline activity of some of my Facebook friends.They consented to it being on this blog post on the condition that it’s anonymous. 

Person 1

Here’s someone’s graph. The x-axis is time, and the y-axis is how online the user is. Possible states for someone’s status are “offline”, “invisible”, “idle”, and “active”. Each coloured line is a different kind of client. It’s called a client because I don’t know I’m an Information Visualisation Professional and I get to make up words like that. Here are explanations for what each of the “coloured lines” means

status - Not sure what this is. Some kind of client-agnostic status? It doesn’t line up exactly with the activity of the other clients though

webStatus - Chat activity on facebook.com

messengerStatus - Status on the Messenger mobile app

fbAppStatus - Status on the Facebook mobile app

otherStatus - Presumably shows when people are online on other apps that can access the API that causes them to be considered “online”. OAuth? Random “apps” like Farmville? No idea

Here’s the same graph, with some clumsy drawings on it showing when I think this person is awake/asleep.

You can see the amount of rest they’re getting each day - it’s the width of the “asleep” bit.

You can also see that they were probably asleep from 3am to 10am on February 11, and BOY does it feel creepy writing this.

Of course, this isn’t perfect, since they might be awake and not using Facebook (I know). Having spoken to a few people who were graphed, it’s been a fairly accurate measure of awake/asleep time, as well as “how much do you browse Facebook at work” time ;)

Do you look at Facebook shortly after you wake up? Shortly before you sleep? If so, these graphs are a fairly accurate way to measure when you were asleep, and anyone you’re friends with on Facebook can do it.

Person 2

I showed this person their graph and asked them some questions.

“Did you go to sleep around 11:10pm last night?” They said yes.

“Did you wake up around 8:32? That’s a weird time. Was your alarm set for 8:30?” They said yes.

NSA APPROVED   ✔️ 🆗👌👌 👍✔️👌🆗🆗👍

Person 3

There are two interesting things about this graph

The person isn’t online as frequently as the previous examples

The person isn’t using the Messenger app nearly as much

You can see that their webStatus was “online” on and off from midnight til around 2am, and then again at 10:21am.. I’m not sure if this spiky pattern means that they really were online, then offline, then online again, or if it’s just a quirk of the dodgy undocumented “API” I’m using, or even if it’s just a problem with my code. 

Similarly, I’m not sure why there are these weird spikes every three minutes (+- ~1minute) sometimes.

Also, why does “otherStatus” go to offline precisely when “webStatus” goes to online? So many questions! Let me know if you know the answers to any of these things (@Facebook employee friends ;) ;) ;))

Anyway, I hope I’ve convinced you that this is real creepy. I don’t really want to be able to have the power to do this.

Your dumb graph screenshots are too small. Give me a live graph to play with

You got it, boss. Click here. Or anywhere, really. This whole sentence is a link. 

What else can you do with this data?

You can aggregate. Finding the average wake up time/sleep time/time spent on Facebook each day and then looking for outliers sure sounds like a way to find interesting things about your Facebook friends.

You can write a thing to email you every morning with the names and sleep times of everyone who’s had less than 6 hours of sleep.

You could even try and guess when your friends are talking to each other, by looking for times when only a few people are active, although I suspect this would be hard.

I’m sure you can come up with something else, too.

Why can you do this? Can’t Facebook stop this from happening?

That’s a good question, thanks for asking.

It makes sense for Facebook to be able to do this, since they can tell when everyone is online anyway. But why can your Facebook friends do this to you? 

I don’t know all the details of how facebook.com uses all the data that’s sent via the /pull endpoint, but it’s kinda creepy that I can see my friends’ status on every device? I guess they could just give me “web” or “mobile” or “offline”, rather than the full list of statuses for every client, but even that doesn’t solve the problem.

I also see the value in seeing “last active 4h ago” and “last active 1m ago” for Messenger contacts but… I dunno, here I am making these creepy graphs. 

Anyway, I just open-sourced my dodgy graph making thing so now everyone can do this. And who knows how many people have been doing it already?

I’m probably oversimplifying it, though. The smart people at Facebook who write this stuff have probably thought of all of this and found that this way was best.

Can I stop you from doing this to me?

Kinda. Coincidentally, because my script is always running, collecting data, I show up as “online” all the time. If you were also running a script like this, it would partially prevent what I’m doing from working on you, since you always show up as “online”, no matter what you’re really doing. Activity from the Messenger app will still show up separately, though.

tl;dr

Facebook sends your computer a bunch of interesting information when you’re on facebook.com. 

You can collect that information over time and use it to keep track of when people are on Facebook, and which devices they’re using.

You can make a pretty good guess as to what time people are going to sleep and waking up

It’s creepy, but I don’t see a way for Facebook to stop allowing this while still making their chat app good.

So how does this make money again?

Oh, no no no. I just uh don’t get out much.

If you want to talk to me about this blog post then I dunno tweet at me I guess. You can also stalk me on GitHub if you want.

For the latest in dumb novelty websites, please direct your browser to http://oneu.se

[saurshaz]

Context matters.

Announcing a bullet train launch in a riot hit village does not strike a chord.

We are all living in a particular context each breath.change is permanent and continuous phenomenon, so is the change in contexts.

A personal story —

I loved the way Pat Rafter played tennis , for those that maybe familiar, he was different in his times. playing serve n volley brand of tennis and winning grand slams on hard courts (back to back). it was ironical. but to me his style of playing seemed unparalleled. I was in awe of this man. He was not the first person to play like that but yet, because of my limited context I felt so.

That was when I started finding more about this style of tennis — ‘serve-n-volley’ (which i enjoyed more over the baseline rallies). I ran into games of Pat cash, Boris Becker, John McEnroe … saw their matches from the archives and watched again and again, and felt like — What beauty !!! Duh , What a fool I was. these guys were bloody entertainers for anyone who loved serve and volleys. Although, I still felt Rafter did something amazing in entertaining Tennis fans like me.

But what I realised from this story of mine was that we all are at all times prisoners of our own ignorance or knowledge. It’s human tendency to think in percentages for our knowledge. we get grades also like that :-) .

But knowledge is not measurable. at least not by someone in flesh and bones.

So, any decision we make is mostly driven by a mix of our gut & our knowledge. I like to think of the knowledge part(along with some other factors I mention below) as `context`. It keeps evolving. there is constant growth in that. So how shall we live a contextually satisfying life (whatever that means) :-)

What we shall aim to do, to lead this limited life in a fuller fashion is to make the contexts at any two points be vastly different from each other always. In other words - expedite the pace of knowledge acquisition in a way that sets stage for much better and more informed decision taking anytime in future than right away. The key is to smartly know about as many things as you can. That smartly is added because you need to let go of things you will never need (as per due diligence). So it’s not random pick whatever you get. You obviously need excitement when you pick knowledge. so you learn about things that interest you and go ahead and learn different things connected to your excitement nerve.

So I was excited about Tennis and I got excited about serve and volley brand of Tennis , that’s why i enjoyed watching an old match of Boris Becker and Stefan Edberg (Pure bliss !!). I might not have enjoyed a similar effort for a similar baseball art, but someone else might have been excited for that. so `pick your flavour of cookie` is important.

Everything is relative !

Context is magically responsible for this too. Your knowledge your circumstances, timings different other factors all determine what something may mean to you.

There is no value in absence of a context

Also, changes in contexts changes the value in the same thing.

What is wrong for me is right for me at some other time

What is wrong for me is right for someone else at this very time

What is success for me is failure for someone else

What was success for me 10 years ago is not succeeding for me now

The differences in above examples is only the different contexts (a mix of time, knowledge, circumstances — i term it as context)

It’s not an easy thing to adopt someone’s context and replace yours with that. I do not think there’s any possibility of any such thing. but just being aware of the fact that what you have as bare minimum is someone’s lifetime dream is a constructive and guiding thought.

In the end i’ll share what I aim for in my life —  I aim for recording the widest possible spectrum of context snapshots in my life.

what that means is that I want to lead a life, which would have had different contexts at different points of time (as different as possible). Can we control that. I think we can do as much as we can.

Cheers !!

[saurshaz]

JS Dev Chatter

Dev 1 : I'm confused mate. Help me with a lighweight, all-equipped, performant Javascript framework.

Dev 2 : Hey I suggest X.js which has awesome data capabilities, but is short in your lightweight requirement. But wait there are C.js and D.js also both of which are highly performant and you would love it, you can use Y.js in addition to them for data requirements(they don't have that). but wait - why don't i right the right combination of those and release those to you as `mostamazingsuperduper.js`. would be cool - right ?

Dev 1 : #$%%^@@@@* **@ *** @@ *@&^@##)(*&@*# !!!!

[saurshaz]

Life in sprints - Lessons from Scrum n Agile

Here’s the introduction — I’m a software man! I learnt to code, i learnt to create mess in projects i worked in, then i learnt how not to create a mess. I learnt to work in better ways. here entered ‘Agile’ & ‘Scrum’ methodologies into my life.

Apart from being fancy names, these methodologies taught me a lot.

I learnt not just about software development. I realised somewhere down the years, that they have lessons for life also.

And I have since become a fan of these methodologies. i have become a regular practitioner of agile and scrum ways to ‘live’ my daily ‘lives’ in a better manner. They have invaded my existence! Software or no software.

became a part of me. just like that :-)

Why the binoculars ? Live on-the-go

Life is short and unpredictable.

I seldom did plan for a distant future. but the few times that i did (long back) , it never did help. when i now look back at that planning act, i see it as a sheer waste of time.

I have stayed that way post scrumagile influence. in fact improved (on my lack of planning too much). Just look the way until next turn and you’re good is the way I now live. Have all the remaining time to actually walk.

A pre-requisite for this approach if you planning to pick this up is no regret policy. it’s an easy one, and if you’re the right kind, you’ll pick this up fast(if not already possessing it)

It’s the on-the-go type. life always has been like that. i wish i could have studied that ways. I want my kids to do that. constantly check your priorities as proceeding along on the journey called life. not regretting. becoming more informed. going by the gut most times, and getting that gut feel better with experiences.

Live the sprints ? What is a Scrum sprint ?

A scrum sprint is a regular, repeatable work cycle in scrum methodology during which work is completed and made ready for review.

Scrum sprints are basic units of development in the scrum methodology. Generally, scrum sprints are less than 30 days long.

The entire purpose of bringing sprints in S/W development as part of scrum was to be in control of the direction, pace and progress towards the overall goal.

Applies to Life too but i would say ‘with a difference’.

So live it per page is what i learnt. living in sprints . Today is a big enough sprint duration for me, with tomorrow and further starting to become tentative for me.

Another thing that sprints emphasise on is ‘focus’. Keep the vision short but do what you are doing with utmost focus. Now that’s what we all are also advised from time we can hear, right ?

But if the peak of the distant mountain is what you need to be looking at always, how do you focus on the way that leads there ?

Therein lies the ‘sprint’. You look at the peak (final goal) divide the journey and then focus on each ‘milestone’.

very much like a relay race. with you running all rounds :-) . difference needs to be that you need to enjoy all the runs , not feel exhausted at the end of each. if exhausted, take a break till you are fine again. A very short life can best be spent happily as much as possible.

Why the rush ? its’ anyways a short tale

This is unlike a S/W development sprint (where there is a rush always)

- You like poetry, find time for it. ditch whatever else you can for that - You like traveling, make an office meeting into one, or a family outing to relatives place one.

Deadlines, resolutions, targets are way too many. you are tired in your mind cursing the management for not understanding their values. But you shall be lenient to yourself. keep soft targets. their absence also ‘may’ hurt.(depends)

Whether you’re a CEO or a executive reporting to the HR manager or an Insurance selling field executive. you will have preferences over the daily job. it will be grossly unfair in your own judgement (sometime later as regret thoughts) if you do not listen to your inner self. so watch the favourite directors’ movies first-day-first-show if your heart says yeah. go out for the rock concert dumping your office laptop at home. lie a little if someone gets in the way of you and you.

you can be a master of living life in a better way. actually ‘only’ you can be.

Agile philosophy for you

So the benefit of Agile was that at the end the client gets what he/she needs (not necessarily what he/she asked for). That is because they are checking constantly on the progress.

if the life is planned in a way like ‘serve in this okay-ish job until the age of 40′ and then startup with doing what i want to ‘always’ do. then there is a problemThis is waterfall planning (the old school method of S/w planning)

you never know what happens next year, you can’t accurately think for the 20 years of your life like that.

if you have a `hobby` to plan. plan in a manner that is visible (or blurred visible)

what if instead of dividing your remaining years of your life into buckets of 30 years(job bucket) and 30 years(aspiration/living my life bucket) you do this -

I live 6 months of my life in mode A and next 6 in mode B

or

I live 1 year of my life in mode A and next year in mode B

or

I live 8 hours of my everyday in mode A and remaining 8 in mode B

or

10–15 days a month in mode A and remaining in mode B

I think i conveyed the point. and it’s not a difficult one. you need to be connecting regularly with yourself on whether the product (your life) is going in desired manner or not.

you might just be very very lucky or courageous and get to live all your hours in single way. you might find a job that lets you do what you want ( tough in my opinion forever)

or

you may fight some years/months to make next years as you ‘desire’ them to be.

the only important point is -

‘do not give in too much to earn to buy something that may not later exist’

constant checking in is a good practice

What are your earning — money or time ?

What are you earning money for ? to buy ‘future’ time ? or making money makes you happy.

if latter — you re in a perfect spot. if not — you are fooling yourself. `present time` in all cases is better than `future time` . use it now. let go some of money for that. At the deathbed i bet anyone would wish a monetary thing that would be related in some way to ‘time’ not ‘money’

The brilliant book — `black swan` has an amazingly well articulated theory around randomness, and the only way to beat that is being indifferent to randomness (latter is my opinion). More we act as being intelligent and understanding the random happenings , more we may look like a fool on a continual basis.

[just-shower-thoughts]

I wish people would stop asking me where I think I’m going to be in 5 years. I don’t have 2020 vision.

[banderboucher]

A message from THE Ray Romano 

[staff]

Have a good weekend, Tumblr. 

[saurshaz]

Hello Web Components !!! (Be Friendly)

How many of you have tried WebComponents ? 

How many of you have tried Polymer ?  What's the first thing that came to your mind in both of the above questions ?

Was it Shadow DOMs. If it was you are pretty much like me. If not, you've either mastered them or are unaware of them, See you soon.

I see templates, I see 2 way data binding, I see custom tags, I see a lot of promise(and a lot delivered too ) when I see Polymer.

What I see most is Encapsulation and I see semantic HTML (courtsey shadow DOm and custom tags)

So, Rod Dodson, Eric beidelman and others have done some great writeups on shadow DOMs , whys and whats.  I would rather wnat to discuss some of the issues, and the solutions that I found for shadow DOM related stuff I've ran into in my journey with web components via Polymer.

Will also discuss each in a following post  1) event tracking is a pain in shadow DOM.  event target does not work, i resorted to much less reliable e.path.(works for my purpose may not in all use cases)

2) cannot get the rendered/composed DOM.  Is there actually a way ? I suppose not, as that will be breaking the encapsulation.

3) Data binding in insertion points(distributed nodes). 

Actually not that big an issue. Create another component and use that in inner content for data binding.

4) Selection using CSS selectors.  I use shadow:: and /deep/ . but not sure they are portable and reliable enough. Are they considered hack. still new on this one.

5) used template binding extensively. once rendered templates are detached.  There shall be a binding between templates and content rendered too :-)

More to come ..