#Hadoop Architecture and Software Development in #Boston – Massive Consumer Data Pipeline

Posted by Steve on July 13, 2015
Posted in: Recruiting. Leave a comment

HadoopInBoston2Boston-area analytics looking to bring on a few more devs to work on the various short and future projects – some of the details are below (any more detail requires a signed NDA or the possibility of death). Roles are senior to team lead – both have architecture elements; openings are because of life changes for folks who have moved on, for people who have been promoted, and for adding to the teams as a result of both new business and for customers requiring substantially more functionality.

The head of software technology is pretty darn easy-going and is a Luke Starwalker wannabe who keeps a light saber handy (but seems to always be in need of new batteries). The company began as a startup, grew, made money, yet retains some startup feel to it (just like adults have the knack for acting like goofy kids when needed). You don’t have to be in the office all the time and if you wear business casual you will likely be met with multiple Golden Retriever head-tilts.

So read on – and LMK what resonates with you (my email is at the bottom). No résumé is needed because we’re just talking…

Details About The Pipeline

  • Pipeline processing utilizes Torque (batch server) and Hadoop
  • Data needed for processing are stored in MySQL and Vertica databases
  • Vertica is used for generating reports used by Data Quality and Data Acquisition; pipeline jobs are orchestrated by a proprietary interface built on top of Luigi; ad-hoc access to pipeline files in HDFS is provided via Hive
  • Initial prep of pipeline files, such as cleansing and removal of sensitive data is via Python scripts and C++ compiled programs; the bulk of the rest of the processing is implemented as Python scripts and Map/Reduce jobs in Hadoop using streaming interface

The Company’s Development Approach

  • Agile and Scrum with 3 weeks Sprints with emphasis on high priority stories (typically 80+% acceptance of stories)
  • Rally project management
  • No special support bucket
  • Scrum and tickets for DevOps

The Software Development Flow

  • GIT for source control; repos are hosted internally
  • Gerrit for code reviews and merges (developer pushes changes in their topic branch up for review)
  • Jenkins for deployment and unit tests (kicks off master merge to Gerrit)
  • Pip install for Python packages, internal PiPy server, everything runs in virtualenv

The Software Development Teams

  • Data Technology – Data Delivery (Pipeline)
    1. Distributed processes, batch processing, Map/Reduce jobs, application of Data Science models for inference and weights generation
    2. Cloud Computing with Hadoop, Torque, MySQL, Vertica
    3. Python, bash, C++
  • Data Technology – Data Mining tools
    1. Development of the tools for Data Miners, mostly CLI
    2. Data Extraction using Hadoop, Torque, MySQL, Vertica
    3. Stronger emphasis on efficiencies and response times
    4. Data Aggregation and metrics generation, application of the weights
    5. Python, bash, Java
  • General Interfaces Team
    1. Interface layers on top of Hadoop, Torque, Vertica, Luigi
    2. Authentication and web services
    3. Application deployment
    4. Python, Java, network/system layers
  • Syndicated Products Team
    1. Python Django, Backbone, jQuery and various JavaScript charting libraries
    2. CSS3, HTML5, LESS
  • Data Collection Agent Team
    1. Browser extensions for IE, Chrome and Firefox, proxy for Windows
    2. JavaScript, C++, Windows Interfaces, Install, Python for reports generation and download software website
  • DevOps
    1. DevOps
      • Production runs of pipeline, backfill, re-runs
      • Pro data publish
      • Hardware troubleshooting
      • Web Servers tuning, Linux in production environment, Windows Servers, IIS, Apache, NFS, SQL, SSH/SCP/SFTP
    2. IT Labs: PC/Laptop support, HelpDesk Change MGMT

Future Tech Initiatives

  • Dynamic Normalization Methodology, Mobile
    1. Complete overhaul of Pipeline
    2. Complete overhaul of Data Mining tools
    3. Redesign of Custom Products: Moving to Dynamic Normalization, simplifying User Experience
    4. Segment Management System: Allow Data Miners to generate, maintain and query a set of standard data segments with automated daily data pulls
  • Next Generation Data Collection
    1. Local Proxy Windows Service
    2. Keeping up with Browser evolutions

As promised, here’s my email (just click it to ping me): levy.steve@gmail.com

Thanks. ~Steve

Recruiting Webinar: Hiring Techs is Easy…Right?

Posted by Steve on July 1, 2015
Posted in: Recruiting. Leave a comment

WebinarHiringTechs

Hardly. Grab the popcorn and listen to my recent presentation on RecruitingWebinars.com: “Hiring Tech Talent Is Easy…Right?”

Click here for the almost hour-long webinar; you can also download my slides from this page?

Any questions?

Hiring Techs is Easy, Right?

Posted by Steve on June 18, 2015
Posted in: Recruiting. Leave a comment

WebinarHiringTechsIt’s not that there’s a shortage of great technical talent – but there does seem to be a significant gap from the sourcing & recruiting professions understanding many things technology.

This webinar addressed 4 core areas around hiring technical people:

  1. The problems sourcers & recruiters have with sourcing, engaging, and recruiting people for technical roles
  2. Understanding the technical mind around work and careers
  3. Identifying specific ways to source, engage, and recruit these people
  4. Implementing better ways to assess the skills of these purple squirrels.
Click here for the entire webinar (about one hour) as well as a link to download the Powerpoint deck ( a new tab will open).
Thoughts?

Trees, Culture, and Recruiting

Posted by Steve on March 16, 2015
Posted in: Culture, Recruiting, Recruiting Association. Tagged: , , . 1 Comment

552380701_snl_church_lady_xlargeSounds as if another ubiquitous #CultureChat has taken place at SXSW replete with recycled lines about what culture is to the experts. Could they all be wrong?

While it’s relatively easy in anthropological terms to talk about how culture is passed on from generation to generation (or management team to management team, employee to employee, customer to customer) via things like storytelling or analysis of clay pot remnants, it’s the initial description of “What is culture?” that vexes almost everyone in HR and talent functions.

Almost everyone speaks of their culture in terms of its uniqueness, how special it is, how different it is from all others – using the same words as everyone else.

Well isn’t that special?

When groups of experts gather ’round to discuss the many shades of culture by using the same language how much learning actually takes place?

I think the reason most use the “standard culture word library” is because accurately describing culture is really, really difficult. It’s this “lapse” in analysis and erudition that leads to the typical “I’ll know it when I see it” action taken during recruiting.

“I’ll know it when I see it but I can’t really describe it.”

How frightening is this? Can you fathom how many great people and performers we’re tossing to the side – and yet at the same time claiming to others that we’re experts at recruiting?

I liken it to the saying, “If a tree falls in a forest and no one is around to hear it, does it make a sound?”; in reality, most everyone inside a company either experiences the culture of the company second hand or takes the word of someone else.

Does the tree make a sound? Of course it must. Is our culture special? Of course it is.

It has to be.

How many in recruiting – especially those newer to it – actually spend significant time inside the groups being supported, with enough time spent observing and collecting data like the great anthropologists? Yes, it’s a rhetorical question.

Instead, most all recruiters simply take the word of their charismatic leader/founder, and/or authentic marketing expert, and/or HR branding guru. Think about how long anthropologists study the remnants of a people who long ago perished before they begin to understand the culture. Yet here in HR and recruiting we take the word of an entrepreneur whose company has been in existence 12 to 18 months?

That’s one of the core problems in recruiting – the forest we think we see through the trees isn’t what we’re describing with our words nor actions. If this isn’t what’s real then I posit that the profession is comprised of lemmings and sheep who need to have their collective eyes and ears examined.

As harsh as the last sentence is, it it spotlights that we’re darn near in the middle of a downward spiraling vortex that’s sucking our collective recruiting reputations down the drain then I’ll deal with the fallout.  It’s time to plug the sink, hire better plumbers, fix the pipes, and enjoy the outcome.

Yes, the falling tree in the forest does make a sound and it’s time to prove it.

Starting now…

[cross-posted on RecruitingBlogs.com]

Spamming The LinkedIn Way

Posted by Steve on March 15, 2015
Posted in: LinkedIn, Recruiting. Tagged: , , . 4 Comments

Since there are so many ways that as recruiters we’re not getting an appropriate bang-for-the-buck from our LinkedIn investments (and more importantly, in any way aligned to the heights of their marketing bluster), I’m going to focus this week on the many ways that we are spammed via the LinkedIn messaging systems.

So to my frenemies in Trust & Safety, this blog’s for you!

For this week, I’ll focus on the Spam. You’ve received LinkedIn InMail that is unequivocally spamful. You scrunched up your face two sentences in and thought, “WTF?” You checked to see what level connection you have with this social Charlatan.

“*(^%$$%@#%#$, first degree.” You right click on their name and Open Link in New Tab…

You furiously scan up and down the profile of this person whom you let into your “trusted network” on LinkedIn. Your brow is beginning to sweat and your eyes are shrinking into beady little pools of molten lava that are ready to erupt. As you’re ready to shout out “DOUCHEBAG”, you click on Contact, because you want to call and skewer this bastard.

Only you see this:

opayq.com?

“Opayq.com is a domain used for Masked Emails for Abine’s MaskMe and PrivacySuite products. These are real user email addresses, created by users to protect their online privacy.”

Lovely.

Now Spammers are gaining control of LinkedIn accounts, adding their masked emails to your account, and sending your contacts LinkedIn InMails like this (I masked a few things myself to protect this person’s identity); notice the punctuation and grammar:

“I give you 100% guarantee of their legitimacy, a trial will convince you” – made me smile, because I was thinking of another kind of trial.

I hope Paul Rockwell has someone on his staff at LinkedIn who will forward him this post.

Fake Profiles, Dating Requests, Marketing InMails, Account Piracy – how soon before LinkedIn ends up as a running joke on TMZ?

And this is the ultimate tool for most recruiters?

[cross-posted to RecruitingBlogs.com]

Yak Away: Maturity and Entrepreneurism

Posted by Steve on March 8, 2015
Posted in: Character, Culture, Education, Entrepreneurism, Ethics, Venture Capital, Yik-Yak. 1 Comment

I’m concerned about the lessons being taught to young technology entrepreneurs:

Create. Grow. Fund. Billionaire.

Personally, I love the concept of entrepreneurism and growth; and who doesn’t have at least one day every year when they wonder what it would be like to be a Billionaire!

But are having a good idea and being able to initially execute on that idea good enough reasons to become a highly sought after target for VC funding?

According to Jim Goetz, a Partner at Sequoia Capital, it sure is – with an interesting caveat: That the young founders of Yik-Yak can magically change the course of online adolescent and young adult behavior.

Jim Goetz, a partner at Sequoia Capital who recently joined Yik Yak’s board, said the app’s history of misuse was a concern when his firm considered investing in the company. But he said he was confident that Mr. Droll and Mr. Buffington were committed to ensuring more positive interactions on Yik Yak, and that over time, the constructive voices would overwhelm the destructive ones.

Jim, is an environment that inadvertently promotes cyberbullying – even if it wasn’t its intended use – a worthy recipient of $62M in Series B funding? Is your confidence that Droll and Buffington – themselves barely few years north of writing the same yaks appearing on their vaunted pre-IPO platform – can magically re-architect Yik-Yak based on some hidden knowledge of human behavior and rid this VC money-printing machine of “the faceless trolls and jerks that abuse these kinds of spaces”?

Jim, what about Yik-Yak’s history as a platform for abuse and cyberbullying didn’t scare you away? Was it the viral takeoff signaling a treasure trove of marketing targets and an IPO payday? What about Yik-Yak isn’t causing you buyer’s remorse? Is it how the app is now downvoting mentions of competitors?

Jim, they can’t be downvoting competitors because Mr. Buffington claims that Yik-Yak is “delivering organic and unfiltered truths, which cannot be said for other news mediums.”

Jim, what about the phrase “unfiltered truth” means downvoting competitors? I must not be very smart to understand this.

For certain, anonymity apps can be used by citizens living in oppressed nations but heck, these countries are already doing a fine job of blocking access. But if you look at where Yik-Yak’s growth takes place – on college campuses, in high schools, the only oppression that takes place are the targets of the anonymous vitriol.

Jim, a platform that by its very architecture and mission offers an easy way to spew this vitriol with impunity. Please Jim, tell me what is so empowering about this – even if it used by a relatively small number of users? Please Jim, tell me how an app like this can help to bolster the self-esteem of young people whose psyche has already taken beatings from members of society who claim that success can only measured by a pair of three-letter acronyms, GPA and IPO?

Jim, social goodness can’t take place without honesty yet the core concept of Yik-Yak – anonymity – veils honesty.

Jim, what are we going to do?

A guilty pleasure used to be a big piece of chocolate cake; in this social media era it seems that proving one’s superiority has eclipsed food as the new dopamine spritz.

Jim, in case you missed this, Iggy Azalea clearly demonstrates she understands elements of human behavior better than you when she wrote that,

4769294610071552

Blue ribbons and IPOs? What about social goodness?

Some lesson.

Jobseekers Taking Chances

Posted by Steve on February 17, 2015
Posted in: #internpro, #jobhuntchat, #OMCchat, Job Search. Leave a comment

[cleaning out my Drafts and I’m surprised I never posted this – originally written December 2013]

This past Black Friday, I watched as Willie Geist interviewed WalMart US “President and CEO” William Simon and asked Bill (actually, Willie tried his best to “pin” Simon against that mythical media “inquisition” wall) questions pertaining to WalMart’s performance, the paying-a-higher-wage issue and future career plans.

From my vantage point, Willie sure seemed to bear his teeth at Bill, pressing him to disclose on public television if (a) he was annoyed at being passed over for the newly available global CEO spot, and (b) if he would raise minimum wages at WalMart. As Willie continued to press, Bill held to the corporate positions but in my opinion, did so in a far softer than the outgoing CEO ever did. Click here for the interview.

At 7:25 AM, right after the interview, I sent off this email to Bill – whom I don’t know – and shared with him my thoughts:

Subject: Your Today Show “interview” – Willie tried but you didn’t budge

Morning Bill-

1. You realize that you’re now the most recruitable CEO target in retail, right?

2. Willie sure doesn’t understand #1 nor does he likely know what succession planning means.

3. Paying above the 50th percentile is all you can ask for as an employee; garnering a higher wage is all about performance – not a Common Law right.

4. People are free to better themselves via a range of education and self-learning channels – leading to potentially higher salaries; Willie might not know this…

5. Good job on the interview – but you should have been firmer with Willie and emphasized #3 and #4. Honesty is a trait that is in short supply in business.

By the way, I know quite a few folks on your recruiting team – good people.

Best regards and revenues,

At 10:03 AM, I received a simple email back from Bill:

Thanks for the kind note.

My message here is not that I reached out to the WalMart US CEO but that I had something to say; rather than keep it between my ears, I elected to act upon it. What’s the worst thing that could have happened?

If you’re a jobseeker seeking to engage a hiring manager or a recruiter trying to engage a passive superstar, what do you notice from what I did?

A customized Subject line?

Some real understanding about your target’s hot buttons?

Statements that support your target’s point of view?

A few sincere platitudes?

Humor and Humanity?

As a jobseeker, would you write something like this? Why – or why not?