Friday, October 4, 2013

'Big Data' doesn't just mean increasing the font size.

Title taken from today's xkcd.

I don't know about you, but I work in an industry that's super excited about "Big Data" right now. I work in Education, but I get the impression that it's not just Education that's interested in Big Data. So, let's start with defining Big Data.

There's a lot of information out there. Most of it is really hard to come to collect and understand. That's why we have scientists carefully creating controlled environments and recording data. That's why we have police detectives collecting evidence. That's why we have auditors leafing through your receipts. But for a while now, we've been accumulating data in electronic form. All electronically collected data is more or less structured.

Usually structured data refers to something like a spreadsheet of numbers while unstructured data refers to something like tweets. But let's expand that out. Even your tweets are somewhat structured. They are made up of typed words, links, and hashtags that can be categorized, collected, and analyzed by a computer. Let's think of them as structured and instead, let's think of unstructured as referring to a behavioral psychologists noting monkey grooming rituals.

A tweet is a direct recording of when and where you tweeted, what you wanted to tweet, how your friends responded to it, etc. Notes on monkey grooming are subjective, possibly missing details and being recorded using inconsistent terminology. The point is, your tweets are ripe for analysis. As is most of your electronic data.

In Education, Big Data includes attendance records, test scores, disciplinary records, socioeconomic status, graduation rates, job placement, etc. In Finance it's everything from stock prices to product reviews to weather reports, to, yes, your tweets. In business it's HR records, budgets, market data, and much much more. I'm hard pressed to think of an industry which isn't collecting structured data.

And that's just the point. We're collecting data. We're accumulating it. It needs to go somewhere and it needs to be stored in a way that makes it easy to access, secure, and safe from degradation. The movers and shakers across most industries have realized this for a while now. That's why we have "data warehouses" and "cloud storage."

Right now is probably a good time to point out that "safe from degradation" is a problem of data collection as much as data storage. Remember when we were discussing how our monkey psychologists might record data using inconsistent terminology? Maybe today our scientist recorded the monkey having, "groomed," but yesterday said, "scratched," and the day before that our scientists said "itched." And maybe two months ago, the scientist switched from taking notes in outline form to using a narrative format. These problems are more common with unstructured data recording, but can also occur in more structured contexts.

Every time these changes happen, the data either needs to be "cleaned" or the analysis of the data needs to be sophisticated enough to gloss over these issues, like how you can ask Siri to, "wake me up tomorrow at 7" and Siri knows to "set an alarm for 7am on Tuesday." That's not exactly what you said, but Siri recognized the intent of your command.

Let's retrace our steps for a moment and identify the data requirements we've encountered so far:
1. Collect the data | Practice good data structuring
2. Store the data | Ensure data is safe, secure, clean

As part of storage, most industries have recognized that you need to access the data once it's stored. Accessing data should be easy and quick. If you need an advanced degree in computer science to pull a report and if it takes two weeks to create the report, it's not ideal. Why isn't it ideal? Because the data isn't useful if you have to call your IT guy every time you want to check on something. And usually the thing you want to check on can't wait two weeks. That brings us to requirements three:
3. Access the data | Ensure it's quick and easy

We've also hinted at requirement four:
4. Make the data actionable | ???

While I'm seeing a lot of collection and storage, I've just started to see my industry realize why they need to access the data. Making data actionable is actually really hard. This is in large part because to really understand data, you need to treat it like a scientist, like a mathematician, like an analyst. It requires identifying variables, setting constants, and running statistical analysis. Sadly, most industries and technologies have barely scraped the surface of this problem. I see a lot of requirements like:
- Automatically show the latest data
- Create bar graphs
- Create pie charts
- Update the graphs and pie charts

And I see very little:
- Run t-test
- Set p value
- Set threshold

Most problematically, I as industry leaders reach towards an understanding of the need for analysis, they jump right to the outcome, skipping how to get there. I see requirements for:
- Early warning indicators
- Graphing of benchmark test scores against high stakes test scores

Do you know what you can tell by graphing two different tests against each other? Very little. It reminds me of this comic. Let's add to number four:
4. Make the data actionable | Ensure analysis is statistically sound

One more time, all four requirements together:

1. Collect the data | Practice good data structuring
2. Store the data | Ensure data is safe, secure, clean

3. Access the data | Ensure it's quick and easy
4. Make the data actionable | Ensure analysis is statistically sound

I think we'll get there, but it might take another decade.

Friday, July 5, 2013

Enough.

Two years and two months ago I did a weight loss competition.  For 10 weeks, my female coworkers and I competed to become the biggest loser.  Each week we weighed in, keeping the weight anonomous but publicizing the losses, and sharing tips and tricks for both diet and excersise.  We helped each other make better lunch choices, and we confessed to each other when we splurged.  Some ladies even worked out together. 

In the end, I was the winner.

It was shocking for me.  I had never paid attention to my weight as a numeric value, and I rarely considered my fitness at all. I felt empowered by my success, but I didn't want to become fixated on a scale or a calorie counter app.  I have too many friends with eating disorders and I carry enough self doubt and stress as it is. 

26 months later, and I'm far heavier than I was even before that competition.  This is probably the heaviest or second heaviest I've been in my life.  I have to buy new clothes every few weeks as I gain more.  I don't want to look at myself in the shower.  I'm scared about my future. 

I've made sporadic attempts at diet and exercise over the past two years, but nothing which was backed by will power.  But two weeks ago, the day after my 26th birthday, as I attended by boyfriend's friend's wedding, I knew I needed to do something.

Even in a flattering dress with a new pair of heels and fashion forward up-do, I found myself hoping eyes would slide over me, remembering my personality but not my looks.  I wanted to hide.  I had enough.

Three days after the wedding, I started a trial membership at a local gym.  In the past 11 days I've gone 10 times.  I stopped going to Dunkin Donuts for a latte/muffin/bagel work day food fix. I stopped suggesting my boyfriend and I order pizza, instead suggesting we roast vegetables.  I can't tell if it's working.  11 days isn't 10 weeks, but I hope that I can achieve the kind of loss I saw before, and then keep it going. 

I have a hard time imagining myself still keeping up this pace in the cold, dark days of January, but I can see myself through to the end of the summer, maybe the end of the fall.  Perhaps by that point it will be habit.

I recently read Children of the Mind by Orson Scott Card.  He introduces an eastern philosophy where in a life of perfect simplicity is achieved not though a continuous monitoring of balance and harmony, but by recognizing the end of endurance and saying, "enough."  A practitioner stays a given course till he or she can take no more, and then shifts directions.  I think of it a bit like a Roomba, waiting until hitting a wall before adjusting course, but in the end, getting the whole room clean. 

That's what happened to me at the end of June.  Enough. I had traveled farther and farther into lethargy until I could take no more. And now I've changed direction.

I plan to take this as far as I can.  To the end of the summer? Until I fit into my clothes from two years ago?  Until I'm at a healthy BMI?  Until I feel comfortable in a two piece bathing suit?  Until I can run a marathon?  I don't know what my cut point will be.  I don't know when I will have had enough.

I'm at the beginning of this process.  With the full range of potential in front of me, I feel optimistic. 

Thursday, April 18, 2013

Interview vs. Ambition

During a job interview, should I lie about what I want to be doing?

It seems like there's an obvious answer.  Of course you shouldn't lie.  Interviews need to be polite, maybe reserved, but you should always be honest about your experience and your skills.  

But everyone you meet wants to fit the square peg in the round hole.  I walk into a interview and I know what they want from me.  They want me to be the job description.  They want me to be the corporate culture and all the other marketing-speak on their website, and they want me to figure out the subtext of our interview and the job description (examples: we need someone organized because the last person was a mess; we need someone with perfect grammar because these documents go to our clients; we need someone who's no nonsense because this role gets very little intrinsic respect).  

They're going to ask about why I'm looking for a job now and what I want to do in 5 years, but I'm not sure they care.  More than that, I think they might prefer not to know.  

It's a dichotomy:  On the one side, most companies aren't interested in hiring someone who has no ambition.  On the other side, no one wants to hire someone who walks in thinking their role is a stepping stone to something more.  Or do they?

I'd hope a good employer would want the person with crazy strong ambition.  As long as that person is also loyal, wouldn't it to be great to grow a customer support rep into a vice president or an administrative assistant into a director?  I've seem team leads say they would be interested in that, but I don't know that it's true.  

I do know that many companies prefer pigeonholing.  They hire you to do X.  Maybe in 5 years you can move to Senior X.  Maybe even X Team Lead.  But what if you strive for the sort of dynamic career that the most interesting and successful people have?  I am starting to feel like no one wants to hire that person. 

Monday, May 28, 2012

If There Was a Grammar Police for Links, I'd Be So Arrested Right Now

Today is Memorial Day!  It's freaking hot here and I believe that holds true for most of the country.  For reasons unknown to me, holiday weekend and hot weather means cooking food on a scorching hot grill under the scorching hot sun in the thick-as-mud humidity.  Yay?

It's also Meatless Monday.  All Mondays are Meatless Mondays.  Even if you aren't an awesome ova-lacto vegetarian like me, you can avoid eating meat for one whole day a week and make an impact in animal rights and environmental issues.  If you follow Meatless Monday, you might see links like this one. It's pretty much the saddest barbecue ever.  Most of their links and tips aren't this bad.  I mean, really?  Salad and grilled veggies aren't most people's idea of a good day.

Here's my idea.

Fruit + Salad.  Not "fruit salad" in the traditional sense, but lettuce, mangos, apples, strawberries, carrots, and anything else you want to throw in.  Depending on the blend of fruits and veggies you throw in, I'd do either a raspberry vinaigrette or an asian peanut sauce.

You could also toss either sauce on extra firm tofu and grill that shiz.

Or throw it on tempeh.  I know, tempeh looks scary and moldy, but it's substantial and holds in flavors very well.

Tofu or tempeh are great as a main course with some grilled veggies on the side, or can be cooled and tossed in your salad.  Hell, wrap it up with the salad for a summer wrap you can take on a picnic.

For desert?  You know what's almost always meatless? Liquor.  Take any leftover fruit and freeze it with some juice and put it in a blender.  Liquor all over before serving.  You know what makes you forget you haven't eaten meat even though you've been grilling all day?  Being drunk.

Okay, okay, you wanted a burger?  Make your own but for the love of god, it should involve black beans and rice, sweet potatoes, or some combination thereof.  Boca Burgers taste gross.

Thursday, March 22, 2012

EA Games and The Sims Social

I promised myself I wasn't going to write about this, but I need to rant.  If you don't feel like reading a rant I'd highly recommend skipping this post.

I love The Sims.  Or maybe it's more accurate to say I used to love The Sims because I'm never playing it again.  I have played The Sims since 7th grade when I went over to my friend Paige's house and she showed it to me.  I've owned several editions of The Sims and countless expansion packs.

I was really excited when The Sims Social came out because I thought it would be a great way to play The Sims with my friends, and it was free!! I hyped it up to my peer group, and even got my boyfriend playing.  And then one day I stopped cold turkey.

The Sims Social was buggy sometimes, quests would be announced but wouldn't show up, my skirt would appear and disappear, floors were missing, accomplishments wouldn't be recorded, but none of that is why I quit.  I quit because one day I finished a quest and my game wouldn't load.  I finished building a room in my house, and this completed the quest.  A box would pop up telling me about it, and as soon as I'd close out of that pop-up, I'd receive an error message.  Every time I logged in the same thing would happen.

After a few days I reached out to EA customer service. At first they were really responsive.  I got a case number and a real human being emailed me (albeit with a form response) and I went though all the basic troubleshooting but to no avail.  Clearly the issue was a bug in the software and not something on my end.

After a few days of emailing, my customer service rep told me he needed to escalate my problem to his manager, but after that I didn't hear back from EA.  I gave it a few days, and then I emailed EA. No response. I emailed them again. No response.  I went on their site and emailed their formal customer service system.  No response.

After two weeks of this I emailed them again, explaining that I was upset with their level of service and that my game still wouldn't move past the error message.  I told them I was quitting the game and would never play or purchase EA products ever again.  And that was the end of it.

The end until today, when ONE MONTH after they stopped getting back to me they emailed me again with a survey and an apology.  A month in a game that rewards you for each day you play?  It's total crap.     I'm more mad than ever and I'm standing by my commitment to boycotting their goods and services. Rant over.

Wednesday, January 18, 2012

Maybe I Have 500 Page Views Because You Can't Look Up Pseudo Problem on Wikipedia Today

Today I reached 500 page views.  Woohoo!

I was planning on writing something momentous when this day came, even though it isn't really that momentous and is actually sort of sad when you think about the number of people who read things on the internet and the tiny fraction of who have glanced at my blog. 

However,

This post isn't actually about the number of page views I have.  It's about SOPA/PIPA.  I'm against these things. I have about a million links about this stuff, here are a few:

http://theoatmeal.com/sopa
https://www.eff.org/sites/default/files/One-Page-SOPA_0.pdf
http://dyn.com/sopa-breaking-dns-parasite-stop-online-piracy/
https://www.eff.org/free-speech-weak-link
http://www.usa.gov/Contact/Elected.shtml
http://blog.reddit.com/2012/01/technical-examination-of-sopa-and.html
http://blog.reddit.com/2012/01/technical-examination-of-sopa-and.html
https://www.google.com/landing/takeaction/

I'm not asking that you take action, just that you be well informed. 

Friday, November 25, 2011

List Series: Pleasant Sounds

A basketball hitting pavement
A shovel scraping against the pavement and through a foot of heavy snow
Small zippers
Cloth rustling against cloth
A knife passing through a juicy apple
My new keyboard with its cover on when I’m thoughtfully typing slowly and without too much pressure
Skin brushing against skin
Coffee beans pouring
Scooping ground coffee beans with a plastic spoon scoop
Water trickling through a small brook or a water feature
The new carts rolling through Target
The fan whooshing noise of the TNG Enterprise
Steady rainfall on trees, wood, plants
Seismic charges in Star Wars episode 2
British people saying "luv" as a term of endearment