Thursday, April 11, 2019

No touching! making calls and SMS with your touch-broken Android phone

the cute Android; trapped behind a broken screen

Crack!

Recently my G9 4G faced an unfortunate (and unspeakable-of) accident; the screen was shattered, and worse, it was rendering upside down (more precisely, vertically mirrored). But even worse, the touch was working upright (upside up)!

To give you an idea of the situation, when I swipe down, the menu unrolls upwards from the bottom of the screen. Then, when I tap the top right icon on the menu, what gets tapped is the icon that appears (upside down, of course) on the bottom right of the screen.

Not a very usable - not to mention user friendly - configuration ;)

Typing without a screen?

As if that wasn't enough, the top two-thirds of the display was completely blacked out - meaning that I could not read anything from the lower two-thirds of the screen; which, incidentally, happens to be where all the keyboard and dialer stuff is.

No way to make a call or type an SMS - "smart" phone, eh?

I didn't have a spare phone or modem/router at hand, so I was literally grounded - muzzled and cut off from the rest of the world.

USB debugging - to the rescue!

Luckily, I had USB debugging enabled; like I always do, as soon as I get hold of a new device.

Frankly, I strongly recommend all of you to have USB debugging enabled on your Androids - authorized for your personal computers. It can save you immense trouble at hard times.

am and service: beauties in the adb shell castle

Once you are in, via an adb shell, you have access to a rich array of Android operational-level commands - most of which work without root access.

am is one of them, which can launch an Android activity (such as the dialer) with desired parameters (yeah, you guessed it; the phone number).

service is another, which allows you to directly invoke methods defined in different Android services, via the command line.

Naturally, my first reflex was to google for "android send SMS via ADB"; which was actually what led me to am and service, and to the actual userland commands.

Sending an SMS with service call: no touching!

Taknok had written an awesome Stack Overflow answer, so I had to look no further.

I did have to dig down the Android 7.0 ISms.aidl source code - because the G9 runs stock 7.0 - in order to decide the actual parameters that I need to pass in. (It was too late when I spotted Taknok's already-written comment on 7.0.)

adb shell service call isms 7 i32 2 s16 "com.android.mms.service" \
    s16 "0911911911" s16 "null" \
    s16 'Yo, ma 4n display z bust. Call ya l8r on a pay 4n! tc' \
    s16 "null" s16 "null"

Here I use i32 2 to send the SMS via my secondary SIM card (G9 is dual-SIM).

Making a call with am: just one touch... or no touching, if you insist!

Again, Stack Overflow helped:

adb shell am start -a android.intent.action.CALL -d tel:"0911911911"

Here there's a bit of a setback, because Android pops up a SIM selection dialog before the call; since my touch was still working fine, I managed to tap the correct SIM card entry with some guesswork. (I couldn't find a way to pass the SIM number to the CALL action, and I suspect it might not be even possible.)

If your touch is totally dead, you could still use adb shell input tap {x} {y} to tap the correct entry - with some trial-and-error.

So there!

You never know when you're gonna lose the display or touch on your Android. Trust me, you never know.

And when it happens, you better be ready.

Bookmark (or earmark, whatever) this post; or the StackOverflow links, if you would rather; enable USB debugging; and wait till the time comes ;)

EDI Exchange: 10 Tips to Pick Your Perfect EDI Provider

communication, collaboration, success - EDI or no EDI

Picking the correct EDI (Electronic Data Interchange) document exchange provider is very important for a trouble-free business workflow; EDI exchange is becoming a very common requirement in today's automated business domain. Things may look complicated, but a simple questionnaire would immensely help you pick and choose between the plethora of EDI providers out there.

Rather than hosting and maintaining your own EDI exchange infrastructure, it is often cheaper and more convenient to bring a third-party (often SaaS, i.e. online/cloud hosted) EDI communication provider.

However, there are a few points that need careful consideration before you do so:

1. What document types are you dealing with?

EDI is just an umbrella term for a wide array of electronic trading document standards: EDIFACT, ANSI X12, ebXML and so forth. Each of these has dozens of types representing actual documents: UN/EDIFACT CUSCAR D.11A for customs cargo reporting, X12 INS-112 for property damage reports, and so forth. In real life you wouldn't _really_ be exchanging EDI, but one or more of these concrete types. So it is important to know whether your provider supports the expected set of document types.

Providers (like AS2 Gateway) are payload agnostic - meaning whatever you give them, they will transfer verbatim; regardless of whether it is an X12 INS-112, an UN/EDIFACT D.11A, or even a JPEG image or a ZIP file. This is very convenient because you can simply plug-in such a provider easily into your EDI exchange workflow, and forget about everything else.

However, if you expect the third-party EDI exchange provider to do additional work based on the document type, you might need to contact the vendor and clarify whether they support it.

2. Do you wish to only send, only receive, or do both?

While most providers do support both, knowing your communication direction would be important. With some high-tier providers, as well as open-source solutions like OpenAS2, configuring the receiving path could be quite challenging; in others like AS2 Gateway, most of the possible receiving configurations would be taken care of by the platform itself.

3. Do you want to modify or transform the document, before transmission or after reception? Or is it just a send/receive?

Some providers offer EDI processing facilities (composing, interpreting and even autogeneration and auto-reply). However, this usually comes at an additional cost and, more importantly, configuration overhead. If you are not looking at modifying or interpreting the content (e.g. your supply chain management (SCM) or warehouse/inventory management system would already be able to do this for you) it would be simple and cost-effective to go for a provider that simply provides EDI transmission capabilities.

4. What is the preferred transmission medium for EDI exchange?

Medium of transmission is independent of document type.

As mentioned earlier, EDI is just an umbrella term, and even UN/EDIFACT CUSCAR D11A is a document type. Usually what is more important is the medium of transfer.

A good analogy is a goods delivery system: you have goods (information) packaged in various ways (formats) - bubble wraps, sachets, boxes, crates, etc. - that are loaded into a vehicle (medium of transport) - van, lorry, freight train, etc. - and carried to the destination.

Similarly, trading information is packaged into various document formats and transmitted to the destinations (trading partners).

A few famous transports used for EDI exchange are:

VAN (Value Added Network)

Contrary to the similarly-named vehicle type, this is a dedicated end-to-end network for exchange of EDI (and other document types) - like privately-owned rail tracks. These are quite expensive and difficult to operate and maintain, so most organizations are moving away from them.

ASn (Applicability Statement) family of standards: AS1, AS2, AS3 and AS4

These are attempts to transfer EDI data securely over existing public networks (like transporting valuables in sealed containers, on public highways). ASn falls into four standards, utilizing email, HTTP (same tech used by browsers and most web/mobile applications), FTP (file transfer protocol) and web services (over HTTP). By far the most popular and widely-adopted is AS2, the HTTP-based protocol.

OFTP (ODETTE File Transfer Protocol)

This is another transfer mechanism like AS2 (more advanced, in some aspects). OFTP is more popular in Europe rather than in the Americas; AS2 is more popular in the latter, thanks to the initiative by Wal-Mart. OFTP has its own network protocol, whereas others like ASx are built on already-established protocols like HTTP/S.

If you are dealing with Wal-Mart, Amazon or any other American partners (large or small), you would probably need to go for AS2.

5. How much do you intend to transfer (say, per month)?

While most providers have extensible plans, they are often quite expensive beyond the basic limit. For example, for a 1000 messages/month package, the cost for 1500 total (500 extra) messages could be significantly higher than the cost of directly going for a 2000 messages/month package.

Additionally, remember that most cloud-based vendors use shared systems for message exchange. If you tend to receive or send batches (spikes) of documents (e.g. every day at midnight) the transmission performance might be affected, especially if the EDIs are large (several megabytes) in size.

bandwidth limit exceeded

6. How many partners are you dealing with?

EDI exchange usually happens between predefined partners - digital identities for easily and reliably communicating with and managing real-world business relationships. If you intend to exchange EDIs with multiple partners, go for a platform and package that supports as many as you want.

7. Do you want multiple trading identities?

Since EDI communication is generally two-way, you as a trading entity will also have your own identity (referred to as station, local station, local partner etc.). Others will refer to you by this identity when they transfer documents to you. The standards allow you to have multiple such identities (e.g. JOHNDOE for dealing with Wal-Mart and JACKSON for Amazon), so if you intend to leverage it, make sure you pick a platform that supports it.

8. Do you have strict security or compliance requirements?

Cloud or SaaS EDI transfer services would often work for generic EDI exchange use cases. However, some trading partners may impose tighter limits; network configurations (VPNs, firewalls and other layers of security), advanced encryption and signing algorithms, non-standard headers, etc.

In other cases, your own EDI/AS2 installation may need to meet various security or compliance requirements; either as specific corporate guidelines or in terms of generic standards like PCI-DSS, SOC or HIPAA. They may be imposed by your partner, or your own organizational policies

In either case, such scenarios may require you to go for a dedicated EDI exchange set-up; or, in extreme cases, an on-premise EDI transfer utility which can then be further customized to meet the specific requirements and compliance guidelines.

9. How would you like to connect the EDI exchange workflow to your existing trade systems?

Manual send/receive may work for up to a few hundred messages per month, but you would need automation if:

  • your message counts are in several hundreds, thousands, or beyond
  • you already have automated trade systems in place - warehouse management, automated order processing, and so forth

On-premise solutions usually offer more flexibility for connecting with existing systems; however, modern online EDI exchange software also offer convenient B2B system integration options. For example, AS2 Gateway allows you to send and receive messages via SFTP (secure file transfer protocol); place your files into a SFTP outbox directory, and read received files via another SFTP inbox directory. Similar entrypoints will soon be introduced for other storage-driven options such as AWS S3.

Meanwhile, REST APIs are becoming increasingly popular as integration points for EDI exchange. REST does offer simple and convenient interaction with general entities; yet, it may be complex when handling EDI payloads; especially when uploading and downloading large EDI exchanges. AS2 Gateway also has a beta-stage API, where you can gain access on demand.

Usually, on-premise EDI exchange solutions can support virtually any type of custom integration. AS2 Gateway On-premise, for example, can connect with a wide array of systems, protocols and transports; including SSH/SCP, files, S/FTP, HTTP/REST, message queues (AMQP, ActiveMQ, IBM MQ, AWS SQS, Kafka etc.), storage-as-a-service like AWS S3, Dropbox, and so forth.

10. What is your implementation deadline - or go-alive timeline?

You cannot run away from deadlines.

On-premise has an installation overhead, while online solutions help you get started with EDI exchange in just a few clicks.

However, setting up communication is the most time-consuming part of the process. Exchanging the connectivity information between the trading parties (e.g. partners in AS2) as early as possible, can greatly simplify and speed up the process.

  • party identifiers (e.g. AS2 IDs)
  • proper message configurations (does each party need their messages encrypted, signed, and/or compressed? if so, with what algorithms?)
  • certificates (encryption, signing, HTTPS, trust chains) and other cryptographic entities
  • any special markers (e.g. custom entries, such as HTTP headers) within the EDI exchange process

Despite the initial overhead, on-premise offers a smoother ride on the long term. Fixes and enhancements can go-alive faster, and you can bring in any desired customizations; even proprietary ones, as long as the EDI exchange vendor can support it. If your trading system is highly dynamic, and evolving and scaling fast, bringing in an on-premise EDI exchange can considerably save time (and money) long-term.

Did we cover all your questions? Do you need to know more? Either way, drop us a line and find the one-stop solution to all your EDI/AS2 needs!

In conclusion

Setting up your EDI exchange may not be trivial. But when you know the requirements and pick the correct provider, the rest would be fairly smooth. Once set up correctly, the EDI exchange would be one of the most automatable, integration-friendly, resilient and maintenance-free components of your trading infrastructure.

Feel free to contact us for advice or guidance in setting up an EDI/AS2 exchange. In most cases, you can follow our documentation and do it yourself as well, on our cloud AS2 Gateway.

Top 5 Reasons Why Your EDI Transfer is Failing

Uh-oh, something's not right with your EDI transfer engine.

EDI transfer is a very popular medium for business document exchange; yet, setting up and maintaining an EDI transfer communication link can be fairly challenging. While SaaS EDI transfer offerings simplify the process to a great extent, there are several pitfalls that need manual intervention; especially during the initial set-up.

Note: This article only applies to EDI transfer via the AS2 protocol. If you are using different protocols, check out the guides for the respective protocol; or consider the benefits of migrating to AS2, the most popular secure EDI exchange protocol.

1. There is something wrong with your network communication link.

AS2 EDI transfer happens over the public internet, so both parties (trading partners - you and your recipient) must be able to connect to each other. As such, this should be your first checkpoint, whenever something has gone wrong.

Is your network up and running?

(a) Network (IP address) whitelisting - you, your partner, or both

Usually applications use network firewall rules for controlling EDI/AS2 connectivity access, so allowing (whitelisting) you on your partner's side would often involve some communication and manual intervention.

Whitelisting comes at two levels:

  • you, on your partner's side (if you are unable to connect/send to your partner)
  • your partner, on your side (if your partner is unable to connect/send to you)

SaaS providers like AS2 Gateway will usually allow access from any network/IP address; and use alternative measures to validate and regulate security. However, if your partner uses a legacy system like VAN (Value Added Network) or an on-premise application, you might need to contact them and send them your EDI application's IP address for whitelisting.

Finding and whitelisting your IP address (when you cannot connect to your partner)

The process of finding your IP address can vary:

  • For a cloud or SaaS installation, you will usually be able to find the IP address via its user interface or documentation; for example, the public IP of AS2 Gateway is 52.204.119.191, as mentioned in their documentation. Contact the vendor if you cannot find the IP address in this manner.

Finding (and whitelisting) your partner's IP address (when your partner cannot connect to you)

Note: This step can be omitted if you go with a SaaS EDI transfer solution like AS2 Gateway which has minimal network restrictions on inbound EDI transfers.

First, check whether the current configuration works; ask your partner to try connecting to your EDI exchange (e.g. send a test message). If it results in an error like "connection refused" or "connection timed out", your firewall is probably blocking him from connecting.

Contact your partner, obtain their EDI transfer IP address (from which he will be connecting to you), and:

  • if you're using an on-premise EDI exchange, whitelist the IP on your organization network/firewall.
  • if you're on a SaaS EDI transfer service, contact the service provider for instructions, settings or assistance on whitelisting the partner's IP address.

(b) You are calling the wrong endpoint on your partner's AS2 system - or the other way around.

Each EDI/AS2 exchange system has a specific endpoint (URL) for receiving incoming transmissions. For example, AS2 Gateway receives EDI/AS2 transmissions at:

  • http://service.as2gateway.com:8280/service/as2-receiver and
  • https://service.as2gateway.com:8043/service/as2-receiver

If your EDI application is not configured with the correct endpoint of your partner (or his with yours), the sending would fail with HTTP errors; such as:

  • 301/302: Redirect,
  • 400: Bad Request,
  • 404: Not Found,
  • 405: Method not allowed,
  • 500: Internal Server Error, or
  • 503: Service Unavailable.

You would need to consult your trading partner and configure his EDI transfer endpoint on your side; and send him the AS2 endpoint of your application so that he can set up your endpoint for transfers from his side.

AS2 Gateway makes this quite easy by allowing you to share your local trading station details with your partner via email, in one click. If your partner is also using AS2 Gateway or the on-premise AS2Station, he will be able to do the same.

2. Your EDI transfer recipient has not properly configured you as one of his trading partners.

In EDI transfer over AS2, each party (sender and recipient) should configure the other a trading partner. This allows both parties to trust each other and utilize various security mechanisms during EDI exchange.

If your transfer is not receiving any response (e.g. causing a "connection closed" error), or resulting in an HTTP error: e.g.,

  • 400: Bad Request,
  • 403: Forbidden,
  • 404: Not Found, or
  • 500: Internal Server Error,

it might be worth contacting your partner and verifying that he has added you as a correspondence on his EDI/AS2 system.

Similarly, if your partner is seeing similar issues on his side, you may need to ensure that you have added him as a trading partner on your EDI exchange application. The steps for defining a new trading partner are usually included in the provider's documentation.

3. You are (or your partner is) using the wrong certificates.

Are your certificates all in order?

Certificates can help to establish secure channels for EDI/AS2 transfers over the insecure, public internet. An EDI exchange involves quite a few certificate types; it is quite possible to swap them incorrectly, or fail to import some of them, resulting in a bad, non-working configuration.

(a) Encryption certificate

This is provided by your partner, and your EDI/AS2 application uses it to encrypt and secure the content being sent out to your partner. (Similarly you need to send your own certificate to your partner, for him to be able to communicate with you.)

This certificate may be missing if your partner does not expect you to encrypt your EDI transfers; in that case you can ignore this (although it would be a security loophole).

(b) Signing certificate

This, also provided by your partner, is used to produce a digital signature; for ensuring that the EDI content has not been tampered with, during the transfer.

It is possible to use the encryption certificate itself as the signing certificate, or to disable signing entirely; in which case you can ignore this step.

(c) HTTPS or SSL certificate

This is used only when the EDI transfer happens over HTTPS; it refers to the certificate that your AS2 application should use for encrypting the overall communication (high-level, not just the EDI content) during transfer.

(d) Chain certificates

Each of the above certificates can be "signed" by a third party, in order to verify that the certificate in your possession truly belongs to your partner; otherwise a third party could send you a fake certificate, and later intercept and read all your communications intended for your partner.

Sometimes this can be a "trust chain" where the original certificate is signed by one party, who is in turn signed by another party, and so forth; until there's a root authority that is implicitly trusted by the EDI/AS2 application, hence validating all the previous trust relationships.

In such cases, your partner may have provided you with one or more "chain certificates" that would constitute this "trust chain", which you would then need to import into your EDI/AS2 exchange application.

If there are no chain certificates, it could mean either:

Dealing with self-signed certificates

While the latter sounds insecure, if you received the certificate file via trustworthy means, it is secure enough for use in EDI/AS2 transfers.

Your EDI/AS2 application may complain that the certificate is "self-signed" and "untrusted" (especially on Windows). In this case, you would need to add it to the list of trusted certificates; but if the application itself has a certificate store feature, it is safer to add it there, rather than at operating system level.

By nature, if you fail to configure any of these certificates (e.g. forget to import the chain certificates); or mistakenly swap them (e.g. use the HTTPS certificate in place of the encryption certificate), EDI/AS2 communications would fail.

Unless your partner has named them properly, it would be difficult to tell these certificates apart. If you know the "owner names" (distinguished names) of each certificate, you can open the files via a certificate viewer and figure out which is which. Most operating systems provide native certificate viewers (simply double-click the file).

Online solutions like AS2 Gateway provide built-in certificate stores for easy addition and management of such encryption, signing and chain certificates.

4. Missing or invalid MDN (receipt notifications)

AS2 protocol optionally enables partners to confirm the receipt of EDI/AS2 transfers via message disposition notifications (MDNs); similar to receipts in monetary transactions.

If the EDI sender has requested an MDN from the recipient, but the recipient fails to provide one, the EDI transfer is considered as a failure. In such cases, AS2 recommends the sender to retry the EDI/AS2 transfer up to a desired number of times.

If the recipient has configured his EDI/AS2 application incorrectly, this could lead to repeated re-transfers of the same EDI content, leading to undesired errors.

Additionally, AS2 allows two types of MDNs. One is synchronous or sync, where the recipient sends back the receipt immediately, as a direct response on the same EDI transfer channel. The other is asynchronous or async, where the recipient briefly acknowledges the original channel, and later sends back a proper receipt on a separate channel. The type of MDN could be explicitly configured at partner level, or requested on a per-message basis using certain AS2 headers.

However, if one of the two EDI transfer applications does not honor this MDN type convention, it could lead to errors and re-transmissions as described earlier.

5. Mismatched EDI transfer configurations

Configurations for EDI transfer? Check!

EDI transfer over AS2 requires quite a few configurations, on your side as well as your partner's side.

Cloud EDI transfer solutions have detailed instructions on these; however the steps are error-prone, and one mistake can lead to complete breakdown of the EDI communication channel.

(a) Double-checking your end

There are a few EDI transfer configurations that you should receive from your partner and configure on your side. Here is a brief summary:

For sending to your partner:

  • Your partner's unique identifier (AS2 ID)
  • The endpoint (URL, address) at which your partner receives EDI/AS2 traffic
  • The type of EDI/AS2 traffic your partner is expecting (encrypted? signed? compressed? if compressed, is it before or after encryption?)
  • For encryption, your partner's encryption certificate
  • With signing, your partner's signing certificate
  • If the transfer happens over HTTPS, the HTTPS certificate of your partner's AS2 software, along with any non-standard signers of it (chain certificates; as described before)

In addition, you can decide whether to request an MDN (receipt) for the sent EDI/AS2 message; and, if so,

  • whether it should be sync or async (see above), and
  • whether it should also be signed (like the AS2 message itself).

While most partners would natively support all these combinations, double-checking could be useful in troubleshooting failures.

For receiving from your partner:

On standard EDI/AS2 applications like AS2 Gateway, this should not usually require additional configuration on your end; other than your partner's AS2 ID. However, the setup may vary based on your EDI/AS2 software.

(b) Double-checking your partner's end:

Here are some EDI transfer configurations that you should send to your partner, and he should configure on his side:

For your partner to send files to you:

This is similar to the set of configurations that you would configure on your side, in order to transfer EDI to your partner:

  • Your unique identifier (AS2 ID)
  • The EDI/AS2 traffic endpoint of your AS2 application
  • For encryption, your encryption certificate
  • With signing, your signing certificate (could be the same as the encryption certificate)
  • If the transfer happens over HTTPS, the HTTPS certificate of your AS2 software; along with any chain certificates

As mentioned before, solutions like AS2 Gateway provide one-click options to share all these details in one click.

For you to send files to your partner:

This should only require your AS2 ID on your partner's end; although your partner's AS2 application may require other configurations as well.

Some applications may require your partner to explicitly define the type of messages (encrypted, signed, compressed, etc.) that you would be sending; others like AS2 Gateway will intelligently derive them from the incoming messages themselves.

Summary

EDI transfers over AS2 could fail due to one or more of a variety of reasons. Checking the underlying error messages is often the key to finding and rectifying these issues.

These issues could be network-level (firewalls and invalid endpoints); content-level (like invalid or swapped certificates); or simple misconfigurations of partner details (AS2 ID, messaging mode or MDN settings) on either your or your partner's side.

Online AS2 providers like AS2 Gateway have support channels where you can obtain further support on any EDI exchange issues experienced by you or your partner.

Saturday, April 6, 2019

BitBucket Geek: Master your Pull Requests on the Command Line (correction: API v2)

BitBucket does have quite a UI these days, but I really don't like websites that update their assets every single day; plus, why all that point-and-click, when you can manage all your BitBucket pull requests via a fabulous API?

Hmm, the API is fabulous; how 'bout the docs?

This was a real bummer. Maybe I didn't have the insight (or patience?) to look in all the right places, but I couldn't find any decent docs on the BitBucket pull requests API v2. (Worth noting that API v1 had some good resources, though.)

Luckily, many had taken my path before me, so the Atlassian forums had some good material to get me started; special thanks to this thread!

The hardest part was pull request creation but I managed to pull it off with some trial-and-error.

Some infra: to call the PR APIs via CLI (wget)

Structured code FTW!

# first, define some shortcuts:

alias dlwg='wget -S --no-check-certificate --content-on-error --header="Accept-Encoding: gzip"'
alias dl='dlwg -O -'
alias dlu='dl --http-user'

function apicall() {
        dlu janakaud --ask-password --auth-no-challenge -q "$@"
}

function confirm() {
        echo -n "Press Ctrl-C to abort $1"
        read
}


# then define some BitBucket-specific shortcuts for patch and PR operations

function bb_patch_op() {
        apicall https://bitbucket.org/api/2.0/repositories/my_org/$1/$2 | gunzip
}

function bb_pr_op() {
        confirm "$3 $1 PR $2"
        apicall --method POST https://bitbucket.org/api/2.0/repositories/my_org/$1/pullrequests/$2/$3
}

You may notice that I am using basic auth (--http-user and --ask-password) for authenticaing to the BitBucket pull request API endpoints. This avoids all the bells and whistles associated with OAuth; besides, you're dealing with life-changing pull requests here, so it actually makes sense to re-enter your password before taking any critical action.

Pull request API calls covered so far

# check the outstanding PRs of a given project; $1=repo
function bb_pr_check() {
        apicall https://bitbucket.org/api/2.0/repositories/my_org/$1/pullrequests
}

# download (to stdout) the patch for a specific PR; $1=repo, $2=PR#
function bb_pr_patch() {
        bb_patch_op $1 pullrequests/$2/patch
}

# download the patch for a specific commit; $1=repo, $2=commit hash
function bb_commit_patch() {
        bb_patch_op $1 patch/$2
}

# approve a PR; $1=repo, $2=PR#
function bb_pr_approve() {
        bb_pr_op $1 $2 approve
}

# merge a PR; $1=repo, $2=PR#
function bb_pr_merge() {
        bb_pr_op $1 $2 merge | gunzip
}

# and, last but not the least, open a PR! $1=repo, $2=title, $3=branch
function bb_pr_open() {
        confirm "opening pull request '$2' on $3 of '$1'"
        apicall https://bitbucket.org/api/2.0/repositories/my_org/$1/pullrequests --method POST --header "Content-Type: application/json" --body-data "{\"title\":\"$2\",\"source\":{\"branch\":{\"name\":\"$3\"},\"repository\":{\"full_name\":\"janakaud/$1\"}},\"destination\":{\"branch\":{\"name\":\"$3\"}},\"close_source_branch\":false,\"reviewers\":[{\"type\":\"user\",\"username\":\"supervisor_1\"},{\"type\":\"user\",\"username\":\"supervisor_2\"}]}" | gunzip
}

For future updates, visit my .bashrc repo.

So there!

So that's how you become a guru that opens, reviews and approves BitBucket pull requests via the CLI. Good luck!

DZone API: the unpublished beast

If you ever want to pull metadata from DZone users or posts, these JSON API endpoints may become handy.

I did not find any documentation for these endpoints, probably because they are intended for use by DZone's internal systems and page rendering logic; so there are no guarantees as to how long these DZone metadata endpoints may remain accessible/unchanged.

Important! I am not endorsing the use of these endpoints by any means; in fact, using them could be a sheer violation of DZone's policies. I am simply sharing my findings. Use at your own risk.

Articles in a DZone newsletter

https://dzone.com/services/widget/sidebar-content-list/DEFAULT?edition={DZone newsletter numeric ID}

Profile details of a DZone user

https://dzone.com/services/widget/users-profile-mini/DEFAULT?user={DZone user numeric ID}

Comments of a DZone post

https://dzone.com/services/widget/content-commentBox/DEFAULT?parent={DZone post numeric ID}

Articles written by a DZone user

https://dzone.com/services/widget/article-listV2/list?author={DZone user numeric ID}&sort=newest

Friday, April 5, 2019

Google Cloud Functions: a (looong overdue) "hello, world!" on GCP

In early 2018, Google Cloud Functions went GA. Some time before that - in March 2018, while it was still in beta - I took some screenshots while taking my first shot at Cloud Functions. But then one thing led to another, and to another, and another, and this blog post was never born.

Now, after a whole year of downright obsoletion, I present to you my "getting started with Google Cloud Functions [beta]" guide.

Surely a lot has changed; beta tag gone, hordes of new features introduced: Python runtime, environment variables, in-built test invocations and logs, and a lot that I haven't even seen yet.

Activate billing on your GCP account

Now, if you rush to the Cloud Functions dashboard, you may notice that you need to enable the Cloud Functions API - unless you have done so already, with your currently active GCP project.

'Cloud Functions' link on the GCP console's left menu

'Cloud Functions API not enabled'

Cloud Functions need a billing enabled GCP project, so that's the first thing we need to do. Google's official guide is fairly easy to follow.

filling in payment details for your billing account

Google promises that we won't be charged during our 1-year, $300 free trial so we're covered here.

Create a cloud function

When done, go ahead to the Cloud Functions dashboard and click Create function.

a fresh Cloud Functions dashboard (back from the beta days)

This will take you to a Create function wizard. The first phase would resemble:

GCP Create Function: configuration phase 1

Here you define most of the basics of your cloud function: name, max memory limit, trigger, and source code.

Later on, you get the chance to define the handler/entrypoint (NodeJS function to invoke when the Cloud Function is hit); and other settings like the function deployment region and timeout (maximum running time per request).

GCP Create Function: name (handler), region and timeout

Trigger options

Triggers can invoke your cloud function in response to external actions: active ones like HTTP requests or passive ones like events from Cloud Storage buckets or Cloud Pub/Sub topics.

In platforms like AWS this may not make much difference: Lambda configures and handles both event types in the same way. But in GCP they are handled quite differently; so different that the cloud function method signature itself is different.

If you try switching between the trigger types, you would see how the sample code under Source code changes:

HTTP functions

Google generates the signature:

/**
 * Responds to any HTTP request that can provide a "message" field in the body.
 *
 * @param {!Object} req Cloud Function request context.
 * @param {!Object} res Cloud Function response context.
 */
exports.helloWorld = (req, res) => {
  // Example input: {"message": "Hello!"}
  if (req.body.message === undefined) {
    // This is an error case, as "message" is required.
    res.status(400).send('No message defined!');
  } else {
    // Everything is okay.
    console.log(req.body.message);
    res.status(200).send('Success: ' + req.body.message);
  }
};

A HTTP function accepts a request and writes back to a response.

The HTTP(S) endpoint is automatically provisioned by Google, at https://{region}-{project-name}-gcp.cloudfunctions.net/{function-name}. So there is nothing more to configure in terms of triggers.

Event-based functions

Google's signature looks like:

/**
 * Triggered from a message on a Cloud Pub/Sub topic.
 *
 * @param {!Object} event The Cloud Functions event.
 * @param {!Function} The callback function.
 */
exports.subscribe = (event, callback) => {
  // The Cloud Pub/Sub Message object.
  const pubsubMessage = event.data;

  // We're just going to log the message to prove that
  // it worked.
  console.log(Buffer.from(pubsubMessage.data, 'base64').toString());

  // Don't forget to call the callback.
  callback();
};

Event-based functions accept an event and convey success/failure (and optionally a result) via a callback.

Here we need to configure an event source (Pub/Sub topic or Storage bucket) to trigger the function.

configuring a Cloud Storage bucket to trigger a function

Good thing is, GCP allows you to pick an existing topic/bucket, or create a new one, right there inside the cloud function wizard page.

defining a new Cloud Storage bucket via the 'Create new bucket' pop-up

defining a new Cloud Pub/Sub topic via the 'Create new topic' pop-up

Picking an existing entity is just as easy:

selecting an existing Cloud Storage bucket

Automatic retry

These non-HTTP functions also have a retry mechanism: you can configure GCP to redeliver an event back to the function automatically, if the function failed to process it during the last time.

Cloud Function retry configuration

This is good for taking care of temporary failures, but it can be dangerous if the error is due to a bug in your code: GCP will keep on retrying the failing event for up to 7 days, draining your quotas and growing your bill.

A function is born

Now we are done with the configurations; click Create.

GCP takes you back to the function dashboard, where the new function will be listed; with a "pending" or "creating" status.

Cloud Functions dashboard, while your new function gets created

It may take a while, but finally you would see the successfully created function in Active status:

Cloud Function created and in 'Active' status

Back in the beta days, I got some sporadic errors when trying to create functions; for no apparent reason. Hopefully nobody is that much unlucky these days.

create function: 'Request failed with unknown error'

It is a good practice to label your functions, so you can find and manage them easily. GCP allows you to do this right from the dashboard.

labelling your Cloud Functions, right from the dashboard

Function actions

Click the three-dots button at the far right end of the function entry, to see what you can do next:

Cloud Function actions

Back in the beta days, there were some hiccups with some of these options (like Copy Function; hopefully they are long gone now!

Copy Function: 'no source code location' error

Testing

Test Function gives a nice interface where you can invoke the function with a custom payload, and view the output and execution logs right away. However it still lacks the ability to define and run predefined custom test events, like in AWS or Sigma. Another caveat is that the test invocations also hit the same production function instance (unlike, say, the test environment of Sigma), so they count towards the logs and statistics of your actual function.

'Test Function' tab of the function detailed view

Test result for the standard HTTP function sample

Logs

View Logs takes you to the familiar StackDriver Logging page, where you can browse, sort, search stream and do all sorts of things with the logs of your function. It takes a few seconds for the latest logs to appear, as is the case with other platforms as well.

'View Logs' takes you to StackDriver Logging

More function details

You can click the function entry to see more details:

  • General tab shows a nice stats graph, along with basic function configs like runtime, memory etc.

'General' tab of our new function

  • Trigger tab shows the trigger config of the function.

'Triggers' tab with an HTTP trigger config

As of now, GCP doesn't allow you to edit/change the trigger after you have created the function; we sincerely hope this would be relaxed in the future!

a read-only trigger config

  • Source tab has the familiar code viewer (although you cannot directly update and deploy the code from there). It also has a Download zip button for the code archive.

'Source' tab with function code

  • Testing tab we have already seen. It's also pretty neat, for something that went GA just a few months ago.

So, that's what a cloud function looks like.

Or, to be precise, how it used to be - back in the pre- and post-beta days.

I'm sure Google will catch up on the serverless race - with more event sources, languages, monitoring and so forth.

But do we need to wait? Absolutely not.

Cloud Functions are mature enough for most of your routine integrations. One major bummer is that it doesn't yet support timer schedules, but folks are already using workarounds.

Plus, many of the leading serverless development frameworks are already supporting GCP!

So hop in - write your own serverless success story on GCP!