Comments by Yuriy Loginov (Top 158 by date)

Still no response from pubs. May need to target additional publishers if still no response by mid next week.

Sending out email to publishers to see if they are ready to run the 300x600 ad on their pages

Every request for terms also performs a write operation which prevents us from using REDIS.

Closing the task

check for "GoogleActiveViewClass” in the AdX HTML to improve AdX detection.

Deployed on: June 24, 2014

tested and deployed, closing task

The code is done and is now being tested. Will deploy tomorrow morning if all is good.

Chris' reply:

1. ID is fine.
2. It just needs to be non-exploit-able

The open rate for the Daily Insider is about 60K per send out of roughly 700K sends. Will keep the open rate tracker active for the Insider to allow us to gauge how effective further mailer changes will be.

The next step is to redesign and rebrand the newsletter.

The remaining 52K subscribers were moved to the Daily Build. Is the next step to suspend the Weekly mailout?

a) how about dmaduid?

b) the reason I chose to use an int instead of a GUID was for faster DB queries since guid would require varchar db field. I can encrypt/hash the int value on the server side so that the client side only sees the hashed value. Is that ok?

Moved another 100K users. The report is still in progress, right now it has Opt Ins, Opt Outs, and Opt In - Opt Out rates. I still want to add Email confirmation rate and current active subscriber metrics.

Out of 722K insiders sent on May 28, 56K were open as of 5:21 PM

I run a sync job with DFP every hour and it previously had no outages on Saturdays.

No reply on the follow up. Lisa and possibly Jeff will continue to follow up.

Started work on the report that will keep track of opt ins, opt outs and total subscribers by mailout. This will help with the remainder of this task will be useful for gauging any other changes we implement in the mailouts.

Added view tracking to the Daily Insider

The opt out rates for the daily build remained relatively stable at 2K a week even after the last move of 100K users.

Moved another 100K users

looking at http://www.reachdynamics.com/ as another ad mailer provider. They provide CPC based text ads that are converted into images and all we have to do is include their tags.

DFP click numbers are now much closer to DM click numbers for geo targeted campaigns. This indicates that the fix is working. Closing the task

DevExpress has an upcoming release and will not be able to share their IP data till mid June. Need to follow up with them about tracking tags so we can start a "test" campaign.

Check out http://www.liveintent.com

They claim to offer real time bidding for newsletter slots

Script for tracking page views and conversion is done. Sent the tracking script to the client and waiting for them to put it up on their page.

Next step is to confirm that the fix is working and that our geo targeting data is accurate. This will require monitoring mailer click data of geo targeted ads to ensure that the recipients match the geo targeted countries.

It appears that two things were contributing to this issue.

1. On the DM side we were allowing the ad to show when the Recipient country was null. This was allowing geo targeted ads to show up for members who's country was unknown. This has been fixed and deployed.

2. On the CP side a large number of members had a NULL CountryID, ~500K. This needs to be fixed as it is impacting mailer geo targeted inventory. For example ~190K out of those 500K are subscribed to the dedicated offers.

Google is not willing to share their data with us. For this exercise we will user mailer click data and potentially CP server logs to match against client data.

In addition to that we are working on a script that will combine all DFP tracking tags into a single JavaScript file hosted on our CDN. This will improve client experience and allow us to make any tracking changes without having to engage the client

The weekly newsletter will be phased out over the course of the next three week. Sales and marketing teams have been notified to no longer sell or advertise the product.

There are 350K remaining subscribers. They will be moved in batches of ~100K over the next 3 weeks.

Batch #1 moved on May 15

report is deployed to production and is now available on apps.DM

The unsub rates for the Daily Build remained stable after moving 50 K users. Waiting for instruction for next phase

It was reported that our ads don't show in IE8. Fixed the problem and deployed the script.

Help powershellmagazine.com get set up with the publisher portal and reporting

Latest move of users to the Daily Build from the Weekly NL did not result in an increase of Daily Build unsubscribes. Will move another 50K users over this week.

Added code that pulls geo targeting data from DFP into DM

I would like to see the following info for banner and newsletter.
Banners/Mailers
--CPM
--Prime Geo
--Non prime Geo
--CPC
--Prime Geo
--Non prime Geo

For each of the above categories the following metrics would show:, impressions, clicks, CTR, eCPM, Revenue

Discuss with Chris whether or not its worth keeping the Weekly NL. Financially it does not get sold very often and we mostly end up running CPC. On the other hand community seems to like it

Another 10K users moved over.

On average we are losing 17 clicks a day by removing ads from online newsletters.

Cross domain tracking now works in IE, in addition to chrome and FF. Safari has a very strict cookie policy and at this point it seems any Safari cross domain cookie solution will be rather elaborate. Some form of fingerprinting may be required in Safari.

Come up with a rate card for Banners and Newsletters. Consider the following criteria:
1. competitor prices.
2. our own network stats

The rate card is to be broken up by CPC and CPM. For CPM show breakdown by banner size. Set dollar amount CPM for geo targeting key locations. Any competitive exclusion or other specific targeting is discussed on case by case basis. The CPC deals are discouraged and should be evaluated on eCPM basis. This requires creative evaluation and then eCPM will be compared against CPM amount.

Comes up with the dashboard that tracks the rate card metrics described above but with no "add on". i.e. it should show raw numbers

DFP ad script return the CreativeID in the response header. We should store it.

Still having issues with DFP moving around the geo location of peer1 which causes issues for geo targeted newsletters.
Need to monitor the effect of removing ads from Web edition of newsletters and evaluate if its worth adding an option on apps.DM to allow the ad to appear in the web version.

The code to randomize the recipient list is live. This option is available on the publication level.
Waiting on the survey design and questions. Once that's ready i can give a demo to Dina to show how the setup works on DFP and DM

Set up our ad words account to link with our DFP account so we can import the Audience data.
See what it takes to get DM listed on AdX as an ad network.

After moving ~22K users from Weekly NL to Daily Build about 8K subscribed back to the weekly NL. No major increase in unsubscribe rates for the Daily Build.

Move another 10K users to the Daily from a different user range

The retargeting tag is now on workspaces. Next step is to set up a DFP campaign that uses the Audience created by the tag.

We can get #1, 2, 3, 5 from DFP.

#4: we have 2,923,886 pages on the network

#7 is available on apps.DM

#6 not sure we can get the number of unique visitiors. We may be a able to get from Google Analytics running on CP but not entire network. We can estimate # of unique visitors based on CP uniques and impressions.

Is this related to the Surveys Dina is doing?

Email sent to Forerunner. Part of the issue lies in that DFP and GA do not track clicks the same way. https://support.google.com/dfp_premium/answer/177507?hl=en

We may need to track clicks on our end or provide a tracking script to client or recommend that clients track campaign id as part of the request URL.

Moving this to lower priority until we hear back from the client and decide if we want to add more tracking.

Step #1. Analyze Forerunner data engage their tech team to find discrepancies

Dave's comments

1. We install our script on a client's site (or more than one) and do end to end tracking, crunch the numbers and find the discrepancies.

2. We create a "test environment" create a mock site (developermetrics.com?), we pay 200 people to follow prescribed viewing patterns (google these keywords, click on the link, google these keywords, click on these adwords, go directly to the site, go do CodeProject click on this ad, etc.) we could use MTurk for this easily. Afterward crunch the numbers and post it as a research paper on DeveloperMedia. I GUARANTEE it would be the biggest promotional piece we ever did. if we did it right it could get international press.

3. Call DFP and have them call Ray @ DevExpress and explain it while we listen in on the call.

4. Call Google Analytics and have them call Ray @ Dev Express and explain it while we listen in on the call.

two groups of users "veteran" and "new" 10K each have been moved over to daily NL. Keep an eye on un-sub rates

A test to run AdX network wide is planned for Mar 26 at 10 AM and will last 24 hours.

Step 1. Test to make sure that the “you have been switched over” message is displayed. First test Mar 25.
Step 2. Once we are sure the message is working, apply the move to a select group of users and monitor stats.
Step 3. Update the moved users tags so they don’t see the “you have been switched over” message after one week.

The low CTR for Daily Build on Mondays was caused by the editorial note on top of the newsletter which pushed the ads below the fold.

Our IP has changed in DFP from Ontario to Manitoba which prevents the mailer from getting the ad. This causes clicks non to track in DFP for geo targeted items. The immediate solution is to add Manitoba manually and automate geo targeting to Manitoba as soon as possible.

the Ad X campaign is set to start March 24 and run until Mar 31st. The daily spend is $35. Revisit the results after 1 week

keyword targeting has proved ineffective. Over a series of tests it has produced a lower CTR rate than non-targeted items

Set up campaign for workspaces on AdX, DFA, or AdWords with about 250/week budget.
1. CP Only.
2. Entire network

The purpose of this is to promote workspaces while also learning how we can gain more visibility on Ad Exchange.

Got the required access from Chris and will start planning the migration process.

Launched cross domain tracking. It is working in Chrome and FF, which covers about 80% of the market. Have to determine why its not working in IE and Safari. Also need to confirm if its working in Opera.

Following up with Google support as to why haven’t received cancelation paper work.

Ran a report for Dave to get news letter CTR by day. Daily build showed low CTR on Mondays. Find out why.

The technical work is done. Next phase would be to study how AdX is performing and implement a feature that will allow user to permanently block given ads.

store referrer string

The non targeted line items for Atalsoft are doing twice as well as targeted ones. Will run a few more tests before concluding key word targeting does not work. Also it would be great to somehow get our hands on the conversion rate data for targeted vs. non targeted clicks.

Can we close this task?

Implement retargeting for workspaces.

1. Create tags which create/remove segment. The tag will also have an optional expiry date.
2. Modify script to recognize these tags and send data to DB
3. Provide Kamil and his team these tags and a quick explanation on how to use them

In order to proceed I would need an account that has read access on the CP database and possibly a crash course in CP database structure. The two tests that I would like to run are:
1. Measure the change in overall unsub rate for the Daily before and after the move.
2. Measure the unsub rate among the users that we moved from weekly to daily.

changed the order progress report to show DFP clicks + 25% for the mailers. At first glance +25% seems a bit much and we may want to decrease that number. Further discussion on this required.

Agreed to use cookies and cross domain iframe communication to establish a global registry of users. Even if Telerik decides not to proceed with Marketo integration we can still make use of global user registry for better Ad targeting.

The report is in place. Follow Eric's guide lines on rolling out AdX to more geo locations

running keyword targeting test for Atalsoft

code to be deployed today

Are we collecting information about missed opportunities? Ad performance, position, etc?

Seems to me our script is in a perfect position to collect fantastic information for us, and information that could save us an awful lot of manual labour.

- are our ads being shown above the fold?
- how many ads are on a page for each publisher
- has the publisher redesigned their page and moved our ad?
- has the publisher added more of our tags or removed any of our tags?
- is the publisher pulling our tags directly or though another system (are we on the page or are we a remnant/3rd party tag)
- etc (I'm sure there are many more scenarios)

This is another big data collection opportunity and so we should err or the site of collecting more data (all if possible).

Also we should get an alert almost in real time when a change occurs somewhere in our network so Carrie can follow up and make sure it's to our benefit.

-TODO: come up with other possible metrics we can track

Provide Dave with schedule on how long it will take to implement. There are two phases to this. First provide a bugging script that will do what Google Audience does, this will save us money. Decide to use cookies or fingerprinting. Phase two is to actually integrate with Marketo through API and Webhooks. Ideally clients should be able to update creative in Marketo, work with Dina and Mike to see if this is possible.

Will work with Katie to run a test to see the effectiveness of keyword targeting. The test will compare the performance of two line items, one targeted one not. Will use AdWords to generate targeting keywords.

All data is downloaded. We may want to save it on our Google drive. Cancel request sent to our Google sales rep. She will send all the paper work to Eric

Follow up with nSoftware to see if they want to try retargeting. This will cost us more, are we willing to absorb the cost?

Created report for viewing down voted Ads. Need to meet with the board to decide if the current set up we have is sufficient to launch AdX in US.

Competitive exclusion was impeding Microsoft delivery. In order to meet Microsoft goal for 300x250 and 160x600 we paused Google, Rackspace and Microsoft 728x90s. This was done because the 728x90 is the first Advert to load on the screen and even though Microsoft had higher priority it was over delivering on 728x90 which then caused Google or Rackspace to take that spot. When Rackspace or Google or Microsoft took that first 728x90 spot we could no longer show the other ad sizes.

V3 don't show ads that are blocked by the user. Store the blocked AdID in a cookie or Web Storage and then determine if the returned ad matches any of the blocked ads.

- Working with Lisa on Microsoft order. Pause Google if Microsoft is under delivering.
- Added more targeted countries to Rackspace order, Dan approved.

Here's is how this could work conceptually.

**Client Page**

---

1. our script generates a fingerprint and reads a Marketo cookie
2. we send this info to our DB { FingerPrint, Cookie, ClientId}

**Marketo API**

---

1. we can call the API directly or client has to expose some of its methods. All leads in the API can be retrieved by the cookie that we stored in the step above. To reduce a number of calls we use getMultipleLeads method which returns 100 leads per call. I imagine the client will not want us to directly query the leads so they will most likely want to expose only part of the data that API returns.
2. we run a job to ID these Marketo leads. Once we get more info on the lead we can start doing retargeting.

**Marketo Webhooks**

---

1. Marketo can do a POST to our service when the lead status changes. This is a huge feature that will allow us to get notifications about all lead activity and the client will have full control of when this happens. Also doesn't cost the client anything but we have to be careful so our servers don't get spammed. Maybe charge per POST?
2. Downside to POSTing like this is that we can't get any unique fields to ID the lead. The best bet is to use I.P and some other fields and then cross reference these against the lead info that we pulled from the API.
3. POSTing will enable the client to tell us what ads they want to server to their lead

**Our Page**

---

1. Fingerprint user and query our DB for lead info.
2. Use lead info to serve an Ad or perhaps queue up an email offer.

**Offer Lead Association service**

---

If the client has no info about the lead but we do, we can offer the client to buy this info from us.

V2 code is live. AdX still disabled in English speaking countries. Need to communicate with publishers and board before rolling out AdX to the rest of the network. Need to set up reports or notifications to quickly react to abusive/offensive ad reports.

In addition to storing keywords we also want to store the full search phrase used to get to specific page. Along with this we also store date and user agent string. The goal behind this initiative is to have data
that we can analyze later.

Later down the road we also want to store the user fingerprint, but this will need to be implemented once we have a reliable fingerprint technology.

Moved from "Optimize banner delivery" task.
Facebook bugging so we can target/insert ads on Facebook using the API.

Need more info to proceed

Closing this task as all the items are defined in their own tasks. Moving the facebook task to its own task

The plan is to run a small scale test as a proof of concept. This will help us decide if we want to invest further into an in-house re targeting solution. nSoftware is a potential client we may try this with. Re targeting in this task refers to using Google Audiences, see Marketo integration for in house re targeting.

V2 Specs:
1. Ad Sense Ads will be decorated with an “X” button
2. Upon clicking the “X” button the ad will be replaced with the following “<undo button="">Don’t like the Ad? Please tell us why <options list=""><report button=""><developer media="" logo="">”. We also fingerprint the Ad so we can chose to block it later if it gets too many down votes or if is reported as abusive.
3. The options list currently includes: Offensive, Abusive, Off topic, Don’t like the Ad. Any suggestions here would really help. I also think we should keep the list of options as short as possible
4. Clicking the “Undo” button will re display the Ad.
5. Clicking the “Report” button will send the selected option to the server.

The decision is to cancel DSM. Cheongs team to download the data from old DSM system. Yuriy to contact Julia to cancel DSM once we have all the data, CC Eric on all communication.

Work with nSoftware to get them more US clicks and ultimately more conversions. The current plan is to split their banner line item into two. One of them specifically targeted to US. Follow up with client that the result is being achieved and make sure this does not impact other campaigns.

We have agreed that we have sufficient US Inventory. Closing task

From Julia @ Google

Question 1 - There are 2 paths for working with DFP:

- Path 1: This is business as usual. If you are using the API to make changes to DFP directly this will continue to function as usual and does not involve Sales Manager at all.

- Path 2: This would be the path of building out Proposals first in Sales Manager and then pushing those changes to DFP. You can utilize the open API that exists for Sales Manager to make those changes, however the biggest thing to understand here is that line item changes will ALWAYS need to originate in Sales Manager and cannot be changed directly in DFP.

Additionally for Path 2 (proposals built in Sales Manager and pushed to DFP), many of the line item fields (dates, impression goals, targeting) are locked down in DFP, meaning that they cannot be changed directly in the DFP UI, but rather all changes must originate in Sales Manager.

Question 2 - The migration process has not started yet and the product team is still finalizing things internally and working on the documentation. The materials and timelines will be sent to you guys as soon as they're available.

Study the benefits of keyword targeted ads vs. regular served ads. The best way to achieve this would be to measure line item performance, one line item targeted the other not targeted for the same ad. Engage clients and let them know we are running these sort of tests in their best interest. Telerik, Red Gate and nSoftware are likely to respond positively to this sort of test.

Do we still need this? Ad Words has an online tool that does just this. Any reason to automate this functionality?

Generate a daily list of non-preferred ad units based on their performance. Group them all into a placement and target adx to that placement. Non preferred pub is under 0.03 CTR and is not on the preferred list. I don't see why we would want to kick some one from the network entirely.

For phase two we need to decide on a simple way for users to report "bad" ads. Perhaps drop dropdown with few options and Send Report hyperlink button. Also include DM logo.

We need a way to distinguish AdX ads that we run through our network (Azure, Google) from true AdX ads. A simple exclusion list of ad fingerprints should work.

Our DFP contact does not return from vacation until the 31st. When I can get an answer on the questions below I will book a meeting with Katie to discuss pros and cons of going to DSM+
1. Are any of the features in DFP+ going to be affected by upgrading to DSM+?
2. Is there a difference between upgrading to DSM+ from DSM and cancelling DSM and then deciding to sing up for DSM+? Do we lose any data this way?

Fingerprinting alone may not be enough to Id the users across our network as it is rather unstable. This past week I ran a test and 4 out of 5 browsers changed their fingerprint at least once through the week. IE was the only browser that did not change its print. To increase our chances of successfully identifying repeat visitors across our network I will try using the algorithm that is tries to guess if the current fingerprint is a descendent of an existing fingerprint. The algorithm is outlined here: https://panopticlick.eff.org/browser-uniqueness.pdf
In addition to that we can also use Web storage or a cookie to identify repeat visitors. Each method has its own benefits

Added line item related fields to the report as per Cheongs request.

We ran a test that suggests that we do have US inventory. The test was to stop serving low paying CPM campaigns with an expectation that it would free up inventory for our higher paying CPM campaigns. We stopped serving Spreadsheet Gear to US on 29th and 30th. We stopped serving Rack Space to U.S on the on 29th and 30th. The expectation was that Google and Microsoft campaigns would get the impressions that were made available by Rack Space and Spreadsheet Gear. Instead the available impressions went to our house Ads which suggests that we had no other U.S ads to serve. The likely reason for this is the restrictive ad serving criteria on Microsoft and Google campaigns which prevents these two advertisers from getting the additional traffic. Google criteria: frequency cap 2/24, US and CAN, desktop only. Microsoft criteria: 1 ad per page, competitor exclusion list, frequency cap 4/24, desktop only, bill on atlas ~15% less than DFP, US only. See attached spreadsheet for the data

I think we can leverage DFP pixel tags to "bug" the client website

You can use Audience pixels to:
•collect users on a page of your site where you have no cookies
•collect users from a third-party site (for example, a landing page of your advertiser)
•collect users who view a particular creative (by embedding the pixel in the creative code)

https://support.google.com/dfp_premium/answer/2508388

Since we are unable to uniquely identify each Ad we will now do the following.
Phase 1. Hide the add and send its info to the server. Notify Ad ops when an Ad reached 1000 down votes or other agreed amount.
Phase 2. Build a web form that allows people to report the Ad as abusive. In cases when we cannot successfully identify the Ad show the form. When we are able to identify the Ad in addition to web form link show the down vote count and a message saying that the ad will be disabled when it hits 1000 votes

The mailer part is already live. The change was implemented in ContentProvider.cs class instead of the mail template. Geo targeted #Research content takes precedence over non geo targeted content