Surviving Terminal Process Death: Zero-Duplicate Payments on Handheld POS Hardware

Deploying mobile point-of-sale software in crowded, high-noise environments, such as stadium concourses, festival grounds, and pop-up retail kiosks, introduces brutal hardware constraints that never appear in web or desktop environments.

Field operators cannot tolerate a 60-second checkout flow. Queue lines demand sub-25 second throughput. At the same time, handheld Android payment hardware operates under intense memory constraints, thermal throttling, and intermittent local socket drops.

During the engineering of the FansChance POS smart terminal platform for busy stadium raffle ticketing, Kaprin isolated and resolved the single most dangerous failure mode in modern EMV card processing: the phantom double-charge.

The Phantom Double-Charge Failure Mode

Most commercial POS architectures rely on synchronous HTTP calls to communicate with local hardware payment daemons (such as the DNA Payments local POSitiveWebLink daemon running on localhost:8080).

The standard, flawed sequence operates as follows:

1. The POS application sends an HTTP POST /sale payload containing the order amount.
2. The payment daemon takes over the terminal screen, displaying the full-screen EMV card-tap activity.
3. While the customer taps their card and the EMV chip negotiates encryption keys with the bank, the terminal operating system drops the background calling socket (ECONNRESET) or kills the calling POS activity to reclaim RAM.
4. The POS application catches a socket timeout or network error upon waking up.
5. The application displays: "Connection Error. Please re-tap card."

The customer taps their card a second time. Both transactions clear through the merchant acquiring bank. The customer has now been charged twice for a single order, sparking angry chargeback disputes and merchant penalties.

The Asynchronous Polling State Machine

To eliminate duplicate charges permanently, Kaprin replaced synchronous HTTP requests with an asynchronous state machine governed by four strict architectural rules:

Kotlin State Machine Implementation

Below is the architectural pattern engineered in Kotlin using Jetpack Compose, StateFlow, and Android SavedStateHandle:

class PaymentViewModel(
    private val savedStateHandle: SavedStateHandle,
    private val paymentApi: PosTerminalService
) : ViewModel() {

    private val KEY_IN_FLIGHT_UTI = "in_flight_uti"
    val paymentState = MutableStateFlow<PaymentUiState>(PaymentUiState.Idle)

    init {
        // Recover in-flight transaction if OS terminated the process
        val savedUti: String? = savedStateHandle[KEY_IN_FLIGHT_UTI]
        if (!savedUti.isNullOrBlank()) {
            resumePollingTransaction(savedUti)
        }
    }

    fun initiateSale(amountCents: Long) {
        val uti = UUID.randomUUID().toString().uppercase()
        savedStateHandle[KEY_IN_FLIGHT_UTI] = uti
        paymentState.value = PaymentUiState.Processing(uti)

        viewModelScope.launch {
            try {
                paymentApi.startSale(SaleRequest(uti = uti, amount = amountCents))
            } catch (e: Exception) {
                // Socket drop is expected during EMV full-screen takeover.
                // Do NOT mark as failed. Immediately transition to polling.
            }
            resumePollingTransaction(uti)
        }
    }

    private fun resumePollingTransaction(uti: String) {
        viewModelScope.launch {
            var attempts = 0
            while (attempts < 30) {
                delay(1000)
                try {
                    val status = paymentApi.queryTransaction(uti)
                    when (status.result) {
                        "APPROVED" -> {
                            savedStateHandle.remove<String>(KEY_IN_FLIGHT_UTI)
                            paymentState.value = PaymentUiState.Success(status.authCode)
                            return@launch
                        }
                        "DECLINED", "CANCELLED" -> {
                            savedStateHandle.remove<String>(KEY_IN_FLIGHT_UTI)
                            paymentState.value = PaymentUiState.Failed(status.message)
                            return@launch
                        }
                    }
                } catch (e: Exception) {
                    // Backoff and retry polling
                }
                attempts++
            }
        }
    }
}

Production Chaos Testing & Field Metrics

To validate resilience before stadium deployment, Kaprin subjected the engine to 150 consecutive chaos injection tests, including hard process kills (am kill) during card taps, simulated loopback socket resets, and forced power cycles.

Summary: Hardware-Aware Systems Engineering

Writing software for physical hardware requires abandoning the cheerful assumptions of web development. Network sockets fail, memory managers kill background tasks, and customers tap cards impatiently. By designing asynchronous polling loops and persisting state boundaries before triggering external actions, engineering teams guarantee deterministic transaction integrity in any physical environment.